Alan engineering

@alanengineering.bsky.social

All things engineering @avec_alan https://medium.com/alan/tagged/engineering

Few-shot examples are only as good as their labels. Ours were hallucinating values from memory. The LLM learned to do the same. Part 2 of the doc processing series: curated datasets, backtesting every change and separating parsing from human knowledge medium.com/alan/lessons...

Lessons from Running an LLM Document Processing Pipeline in Production

We evolved from text-only to multimodal, built curated reference datasets for RAG and developed an evaluation framework

medium.com

Ever wondered how Alan reimburses health claims so fast? Spoiler: we automated 70% of document processing. We just dropped the first article of our document processing series — stay tuned for more on how we're scaling this across other countries and use cases. buff.ly/2Lu7aWD

How we reached 70% document processing automation at Alan

We replaced our classic ML pipeline with a unified LLM-based approach using Markdown Transcription and few-shot RAG extraction

medium.com

"Claude did it almost on its own while I was boiling water for my tea" That's how our engineers now fix flaky tests at Alan. Read more about our journey from manual debugging to AI-powered investigation in our latest article 👇 medium.com/alan/from-gu...

From Gut Feeling to AI assisted: How We Automated Flaky Test Investigation

Over the last year, we’ve improved our flaky tests investigation process, moving from manual debugging to an AI first solution. This…

medium.com

There are many LLM benchmarks such as MMLU and GSM8k, but they're useless for AI agents. Real agents need to handle database state, tool calling, and multi-turn conversations. Stateful benchmarks show the path forward. New post on agent evaluation 👇

Benchmarking AI Agents: The Challenge of Real-World Evaluation

AI agents need stateful benchmarks. Unlike LLMs, agents interact with databases and users. We explore why and how to evaluate them…

medium.com

Static chatbots couldn't handle complex support tickets about insurance claims. So we built something different with tool calls and the ReAct framework. Our Claim Agent investigates dynamically - just like human agents, but faster. Now automating 30% of tickets it receives.

Bild