Moved a demo agent to a 3x smaller model. Replies read fine. The diff showed it had stopped calling the cancellation tool entirely — while still saying "your subscription is cancelled." That's what whatbroke catches: behavior drift text diffs can't see. github.com/arthi-arumugam-git/whatbroke

ran experiment 2 with whatbroke: swapped llama3.2:3b for qwen2.5:3b in the same support agent. same size class, same prompts, temp 0.2. the diff came back with 12 changed findings. last week's 3x size downgrade only produced 3.

whatbroke 0.3.0 is out. Two things people kept asking for: import your existing traces (Langfuse, LangSmith, OpenTelemetry GenAI spans) instead of re-instrumenting, and --fail-on breaking for CI. github.com/arthi-arumugam-git/whatbroke

whatbroke 0.3.0 is on npm. New: whatbroke import converts OTel GenAI spans, Langfuse and LangSmith exports into diffable traces, so you can diff runs you already recorded without touching your agent code. Plus --fail-on for CI exit codes. github.com/arthi-arumugam-git/whatbroke

I swapped a tool-calling support agent from llama3.2:3b to 1b. Same scenarios, same prompts, one string changed. Then I diffed the trace files to see what the cheaper model quietly changed. 0 breaking, 3 changed, 12 info. The interesting bits are in the changed ones.

I kept getting bitten by silent regressions every time I swapped models under an agent. So I built whatbroke: record two runs of a tool-calling agent, diff them, see exactly which tool calls and args changed. MIT, everything stays local. github.com/arthi-arumugam-git/whatbroke