Towards Data Science
@towardsdatascience.com
The world's leading publication for data science and artificial intelligence professionals. Website đ towardsdatascience.com Submit an Article âïž https://contributor.insightmediagroup.io Subscribe to our Newsletter đ© https://bit.ly/TDS-Newsletter
Follow along Shuai Guo's new tutorial to learn how to build managerâspecialist workflows with the OpenAI Agents SDK.
Using Agents as Tools | Towards Data Science
Building managerâspecialist workflows with the OpenAI Agents SDK
towardsdatascience.com
"A context compiler does not make a codebase smaller, but it makes passing that code to an LLM practical by filtering for what actually belongs in the prompt." Emmimal P Alexander's new deep dive unpacks the process of building a context compiler for coding agents.
Coding Agents Donât Need Bigger Context Windows â They Need a Context Compiler | Towards Data Science
Most coding agents treat prompt construction like retrieval: gather more files, add more context, hope the model figures it out. But that approach breaks down fast. As context grows, irrelevant codeâŠ
towardsdatascience.com
In a clear and focused guide, Shuai Guo introduces us to a hybrid LLM application pattern that combines a predefined workflow with adaptive agent behavior
Put the Agent Inside the Workflow | Towards Data Science
A hybrid LLM application pattern that combines a predefined workflow with adaptive agent behavior
towardsdatascience.com
In a practical tutorial, Abdullahi Dattijo explains how to record model tool requests, real function results, patches, checks, and more, so you can debug coding agents' outputs when you need to.
How to Debug AI Coding Agents When They Change the Wrong Thing | Towards Data Science
A practical tutorial for recording model tool requests, real function results, patches, checks, screenshots, and a saved run log.
towardsdatascience.com
Ready for a whirlwind journey through math, optimization, and operations research? Don't miss Luis Fernando PĂ©rez Armas's new deep dive on how Benders decomposition works.
How Benders Decomposition Works Part I: Optimality Cuts | Towards Data Science
A friendly introduction to one of the most powerfull optimization techniques using the uncapacitated facility location problem
towardsdatascience.com
How will AI reshape upper management? Anubhab Banerjee explores a potential future when agents might become sharper coordinators than humans.
When the Code Becomes the CEO: Why Your Next Manager Might Be a Decentralized Agentic Loop | Towards Data Science
In five to ten years, the sharpest manager in your company might not be human, might not sleep, and might exist entirely in shared GPU memory. This is the systems-level view of the algorithmicâŠ
towardsdatascience.com
For his latest deep dive, Sean Moran explores time-series LLMs, unpacks their inner workings, and tests their performance.
Time-Series LLMs, Explained with t0-alpha | Towards Data Science
t0-alpha is a decoder-style patch transformer for probabilistic time-series forecasting. Raw series are split into 32-step patches, embedded, processed through causal time-attention andâŠ
towardsdatascience.com
Revisiting the topic of question-parsing in RAG systems, Angela Shi and Kezhan Shi share six concrete insights that go against the mainstream RAG playbook.
The Untaught Lessons of RAG Question Parsing: Structure Before You Search | Towards Data Science
Enterprise Document Intelligence [Vol.1 #6ter] - Six positions on the question-parsing brick that contradict the mainstream RAG playbook
towardsdatascience.com
If you want to increase the effectiveness of your coding agents, Eivind Kjosbakken recommends implementing end-to-end testing, and explains how to do that using Claude Code.
How to Run End-to-End Tests with Claude Code | Towards Data Science
Increase the effectiveness of your coding agents through end-to-end testing.
towardsdatascience.com
What's going on under the hood when your RAG system validates a generated answer? Kezhan Shi looks at the inner workings of a crucial component in the process.
Validating the RAG Answer Before the User Sees It: Spans, Quotes, and the Feedback Loop | Towards Data Science
Enterprise Document Intelligence [Vol.1 #8C] - Structured output is the start of validation, not the end: check the evidence, accept not-found, loop the feedback
towardsdatascience.com
Give agent teams a cleaner way to decide which configs to ship, prune, and route toward next â Doster Esh presents several potential approaches you should add to your toolkit.
Stop Ranking Agent Configs by Average Score | Towards Data Science
Best-worst comparisons, MaxDiff-style judging, and Plackett-Luce utility scores give agent teams a cleaner way to decide which configs to ship, prune, and route toward next.
towardsdatascience.com
Zooming in on the generation step of RAG systems, Kezhan Shi outlines the benefits of a customized dispatcher that turns a parsed question into a typed LLM call.
Assemble Each RAG Generation Prompt from a Base Prompt Plus the Rules Each Question Needs | Towards Data Science
Enterprise Document Intelligence [Vol.1 #8B] - A fixed BASE, the rules each question needs, one registry: the dispatcher that turns a parsed question into a typed LLM call
towardsdatascience.com
After outlining the building blocks of an enterprise-grade RAG system, Angela Shi explains what it takes to upgrade each of its key components.
A Production RAG Pipeline for PDFs: Relational Parsing, TOC Retrieval, Typed Answers | Towards Data Science
Enterprise Document Intelligence [Vol.1 #9A] - Same paper, same question as Article 1. One upgraded contract per brick: document parsing, question parsing, retrieval, generation
towardsdatascience.com
When should you use RAG and when is it time to turn to fine-tuning? Maria Mouschoutzi shares an accessible guide to two powerful approaches and the tradeoffs they present.
RAG vs Fine-Tuning Explained: What They Actually Do and When to Use Each | Towards Data Science
Two techniques, two different problems, and why the question is not really "which one wins"
towardsdatascience.com
Ready for a thoroughly engaging deep dive on using math to solve thorny routing problems? Don't miss Luis Fernando PĂ©rez Armas's latest, based on his own industry experiences.
âLos Movimientosâ: The Routing Problem That Nearly Broke My Spirit | Towards Data Science
Using mathematical optimization to solve a pickup-and-delivery problem with time windows.
towardsdatascience.com
Curious to learn about active learning and how it can help you optimize â and reduce â the time you spend annotating training examples? Lucas Braga shares a comprehensive introductory guide.
Reducing Human Annotation with ML Active Learning | Towards Data Science
In a world where human time is expensive, learn how to use it only when really necessary
towardsdatascience.com
We're thrilled to share Justin Stewart's first TDS article: an in-depth analysis of the real costs of running LLMs locally, down to the Watt.
How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon | Towards Data Science
Five models, sustained generation, real wall-socket energy at $0.31/kWh â and the surprise the RTX-3090 numbers predicted, only bigger.
towardsdatascience.com
In her new article, Sara A. Metwalli presents an actionable introduction to Hugging Face, an essential (open source) tool for data and ML practitioners regardless of their specialty.
The Python Ecosystem That Changed AI Development | Towards Data Science
How one open-source ecosystem made state-of-the-art AI accessible
towardsdatascience.com
If your coding-agent setup feels a bit messier than it should be, don't miss Eivind Kjosbakken's guide to organizing your tasks in an orderly and streamlined manner.
How to Organize All of Your Coding Agent Tasks | Towards Data Science
Optimise how you interact with your coding agents
towardsdatascience.com
"The Jacobian Conjecture asks whether a particular type of function must always be invertible. The disproof of the conjecture is simply a counterexample: a function that meets all the conjectureâs requirements but provably cannot be inverted." James OâBrien presents an accessible explainer.
A Simplified View of the Jacobian Conjecture | Towards Data Science
The full conjecture is stated over abstract fields, but the counterexample is a concrete 3D function that we can explain and visualize using familiar geometric ideas and a little algebra.
towardsdatascience.com
In his latest exploration of the analogous relationships between physics and AI, Spyros Georgopoulos digs into the inner workings of LLMs' temperature parameter.
How to Decode the Temperature Parameter in LLMs | Towards Data Science
How statistical physics explains the transition from deterministic predictions to generative AI.
towardsdatascience.com
What does it actually take to turn a company's scattered knowledge into an LLM-ready context layer and company "brain"? Follow along Tomer Mesika's practical insights to find out.
How to Build a Context Layer and a Company Brain | Towards Data Science
What it actually takes to turn a company's scattered knowledge into something an LLM can reliably use â and why the demo is 5% of the work.
towardsdatascience.com
Not sure what model context protocols are, or whether you should incorporate them into your agentic workflows? Maria Mouschoutzi provides an accessible primer covering custom integrations, tool access, and more.
MCP Explained: How Modern AI Agents Connect to the Real World | Towards Data Science
from custom integrations to a universal standard for tool access
towardsdatascience.com
Don't miss Emmimal P Alexander's enlightening account of a production incident, a 150-line static analyzer, and the case for treating prompt changes like schema migrations instead of string edits.
Prompt Engineering Is SolvedâPrompt Management Isnât | Towards Data Science
Prompt engineering helps you write better promptsâbut it doesnât help you change them safely. This article explores a common production failure where a simple variable rename breaks every live call,âŠ
towardsdatascience.com
In an incisive new article, Ananya Bhattacharyya explains why prediction-driven variable selection misses confounders, and how Bayesian Adjustment for Confounding attempts to fix it.
Why Your Best Predictive Model Gives the Wrong Treatment Effect | Towards Data Science
Why prediction-driven variable selection misses confounders and how Bayesian Adjustment for Confounding attempts to fix it.
towardsdatascience.com
In the second part of his top-notch series on solving pickup-and-delivery problems, Luis Fernando PĂ©rez Armas explores the potential of adaptive large neighborhood search (ALNS).
Los Movimientos, Part II: Solving Large Pickup-and-Delivery Problems with Adaptive Large Neighborhood Search | Towards Data Science
Building an ALNS heuristic in Python for vehicle routing, time windows, capacity constraints, and mandatory driver breaks.
towardsdatascience.com
We're thrilled to welcome Rahul Saha to TDS â don't miss his comprehensive guide to avoiding entity key drift in data lakes.
Avoiding Entity Key Drift in a Data Lake: Step 1, Normalization | Towards Data Science
This is the opening piece of a four-part deep dive series, on building a high-frequency streaming pipeline against a live public API. The data source is openSenseMap, a citizen-science IoT networkâŠ
towardsdatascience.com
"You may be able to get away with it, as it provides a sensible default, but in the long run, especially when encountering a non-stationary or difficult optimization landscape, you will fail. Hard." Sam Black explains why you shouldn't treat the Adam optimizer as an automatic go-to.
Donât Just âThrow Adam at Itâ: Misunderstanding Adam Will Cost You | Towards Data Science
You "vibe coded" the import. Understand Adam's optimization dynamics, why it fails spectacularly, and how to fix it.
towardsdatascience.com
Interested in gaining a deeper understanding of how backpropagation works? Nikhil Dasari continues to explore the inner workings of this foundational algorithm in part 2 of his series on the topic.
Backpropagation Explained for Beginners (Part 2): There Has to Be a Better Way | Towards Data Science
The idea that makes backpropagation possible.
towardsdatascience.com
When's the last time you spared a moment to think about the "humble" mean? Sachin Date presents a meticulously researched and mathematically robust history of (arguably?) the most useful of all statistical concepts.
The Most Beautiful Statistic: The History and the Science of the Humble Mean | Towards Data Science
The mean keeps making its usefulness felt in all sorts of situations, often in truly non-obvious ways
towardsdatascience.com