We built a thing! The Databricks Reranker is now in Public Preview. It's as easy as changing the arguments to your vector search call, and doesn't require any additional setup. Read more: www.databricks.com/blog/reranki...

Reranking in Mosaic AI Vector Search for Faster, Smarter Retrieval in RAG Agents

Boost RAG agent quality with reranking—deliver more relevant answers in less time with a single parameter in Mosaic AI Vector Search.

databricks.com

Excited to launch Agent Bricks, a new way to build auto-optimized agents on your tasks. Agent Bricks uniquely takes a *declarative* approach to agent development: you tell us what you want, and we auto-generate evals and optimize the agent. www.databricks.com/blog/introdu...

Introducing Agent Bricks: Auto-Optimized Agents Using Your Data

Discover Agent Bricks by Databricks — a new way to build production-ready AI agents using your data. Automatically evaluate, optimize, and scale agents with higher accuracy and lower cost.

databricks.com

Apache Spark 4.0 is out with some huge improvements across the board. SQL’s much more powerful, Spark Connect makes it easier to run apps, new languages and more. It’s amazing to see the community still growing fast and releasing over 5000 patches in 4.0. www.databricks.com/blog/introdu...

Introducing Apache Spark 4.0

Explore Apache Spark 4.0's key updates: advanced SQL features, improved Python support, enhanced streaming, and productivity boosts for big data analytics.

databricks.com

#MLSys 2025 is next week! You can still register at mlsys.org.

Nice results on never-ending learning for code editing. We believe that a lot of AI applications will be customizable this way (to every company's codebase, users, etc). The combined AI serving, data and MLOps environment on Databricks makes these easy to build. www.databricks.com/blog/power-f...

The Power of Fine-Tuning on Your Data: Quick Fixing Bugs with LLMs via Never Ending Learning (NEL)

Discover how fine-tuning small open-source LLMs on interaction data enables faster, cheaper, and more accurate code fixes with Databricks Quick Fix.

databricks.com

🎥 New Video: Get Hands-On with MLflow Tracing! In this video, @danliden.com walks through how #MLflow Tracing boosts observability in #GenAI apps—great for debugging, experimentation & organizing data workflows. Watch now ➡️ www.youtube.com/watch?v=iRbB... #opensource #oss

MLflow Tracing | Introduction & Tutorial

YouTube video by MLflow

youtube.com

Really cool result from the Databricks research team: You can tune LLMs for a task *without data labels*, using test-time compute and RL, and outperform supervised fine-tuning! Our new TAO method scales with compute to produce fast, high-quality models. www.databricks.com/blog/tao-usi...

TAO: Using test-time compute to train efficient LLMs without labeled data

LIFT fine-tunes LLMs without labels using reinforcement learning, boosting performance on enterprise tasks.

databricks.com

The #MLSys2025 program is up and registration is open! Check out accepted papers at mlsys.org/virtual/2025... and sign up to attend at mlsys.org/Register.

Exciting news—MLflow 2.21.0 is live! 🎉 This release includes significant features, enhancements, and bug fixes to improve documentation, #GenAI prompt management, tracing & more. 🔗 Explore all the new features & improvements: mlflow.org/releases/2.2... #opensource #oss #mlflow

🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs! We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.

We're probably a little too obsessed with zero-shot retrieval. If you have documents (you do), then you can generate synthetic data, and finetune your embedding. Blog post lead by @jacobianneuro.bsky.social shows how well this works in practice. www.databricks.com/blog/improvi...

Improving Retrieval and RAG with Embedding Model Finetuning

Fine-tune embedding models on Databricks to enhance retrieval and RAG accuracy with synthetic data—no manual labeling required.

databricks.com

We're bringing in a new era of enterprise data management and agentic AI with SAP Business Data Cloud with Databricks. ✅ Unifies your SAP and non-SAP data ✅ Natively embeds Databricks technology ✅ AI agents streamline workflows Learn more: sap.to/sapbdc

Sponsor registration is open for #MLSys 2025. We have the most submissions ever to MLSys so it promises to be a great conference! mlsys.org/Sponsors/spo...

Researchers open source Sky-T1, a ‘reasoning’ AI model that can be trained for less than $450

Researchers open source Sky-T1, a ‘reasoning’ AI model that can be trained for less than $450

So-called reasoning AI models are becoming easier — and cheaper — to develop. On Friday, NovaSky, a team of researchers based out of UC Berkeley’s Sky Computing Lab, released Sky-T1-32B-Preview, a reasoning model that’s competitive with an earlier version…

tcrn.ch

"Sky-T1-32B-Preview, our reasoning model that performs on par with o1-preview on popular reasoning and coding benchmarks." That was quick! Is this already the Alpaca moment for reasoning models? Source: novasky-ai.github.io/posts/sky-t1/

Congrats to Meta on releasing Llama 3.3, a 70B model that matches the performance of Llama-405B! Open weight models are advancing so rapidly and the cost to get this performance is quickly going down. We're thrilled to let users serve & customize this on Databricks. huggingface.co/meta-llama/L...

meta-llama/Llama-3.3-70B-Instruct · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

Compound AI Systems, Inference-time Compute Meetup @ NeurIPS 2024, with many AI luminaries as panelists. Poster submissions are open: lu.ma/q5r8b67t

Compound AI Systems, Inference-time Compute Meetup @ NeurIPS 2024 · Luma

Meetup for practitioners and researchers working on and interested in compound AI systems, inference-time strategies and scaling laws, networks of networks,…

lu.ma