Tracing an LLM app is a cost tool too. Token usage per model, per user, per feature means you optimize spend instead of finding out at the invoice. One span in our example burned 8,814 tokens. go.upgradejs.com/umw #LLMOps
🤖 OmbuLabs.ai
@ombulabs.ai
Your Data Holds Untapped Potential We can turn it into a competitive advantage 🚀 We help organizations uncover hidden opportunities in their data and build AI-powered solutions that drive real impact. https://ombulabs.ai
Incompatible Rubies, three version managers, bundle install dying on a C extension. Containers skip OS emulation and lean on chroot, namespaces, and cgroups, which is why one boots while a VM is still allocating RAM. go.ombulabs.com/9oi #LegacyCode
Jev returns only a typed answer: a Choice, a Score, or a probability. No free text means it cannot hallucinate an option you never defined. go.upgradejs.com/46s #AIEngineering
Judge panel unanimity in agent evals hit 18.13x false consensus lift on one evidence type. Agreement was correlated error, not independent validation. go.upgradejs.com/r18 #AIEngineering
Ask a narrow question, not a mission. Jev says its 0.8-probability answers land right about 80% of the time, and it will not invent an option outside your schema. go.upgradejs.com/g1u #AIEngineering
C2PA metadata says which software made a file, not who made it. And screenshot a marked image and the manifest is gone entirely. Worth knowing before you build policy on it. go.upgradejs.com/shc #AIPolicy
Step-level agent grading caught 91% of injected faults at hop one, then zero at hop two. Later steps quietly patch earlier mistakes. go.upgradejs.com/sv9 #AIEngineering
An agent is a loop, not a chatbot. Reach for one only when a task is frequent, too varied for fixed rules, and safe enough to get wrong sometimes. go.upgradejs.com/395 #AIAgents
Skip the 1-10 rubric. Binary pass or fail gives an LLM judge a cleaner signal, and it scales the same human calibration that built the eval set. go.upgradejs.com/oci #LLMOps
Sandi Metz's limits are blunt and useful: classes under 100 lines, methods under 5, no more than 4 parameters. Ask a class what its job is, and if the answer needs an "and", you found the split. go.upgradejs.com/7oh #Refactoring
String interpolation inside a where clause is the classic Rails SQL injection. Pass the value as a bind parameter instead, then let Brakeman check the rest of the app. go.upgradejs.com/7ft #RubyOnRails
A Rails 4.2 app nobody wants to touch is a good candidate for a container. Start from Ubuntu, set DEBIAN_FRONTEND=noninteractive so Postgres stops prompting mid-build, and the whole setup ritual turns into a file. go.upgradejs.com/8lc #Docker #Rails
Give retrieval its own span: the query you sent, the documents that came back. Then a bad answer stops being a mystery and becomes one question, retrieval or generation. go.upgradejs.com/y86 #RAG
We rebuilt our case studies page. Same four projects, far easier to read now. www.ombulabs.ai/case-studies...
Sort your AI tools into three buckets: annoying if it dies, slow if it dies, revenue stops if it dies. Only the third needs a backup provider now. go.upgradejs.com/iga
An LLM judge earns its keep once it agrees with your human labels 80 to 90 percent of the time. Below that you are automating noise. go.upgradejs.com/8nk
Pick one sanctioned low risk use case and let the whole team practice there first. Permission plus a boundary beats a policy nobody reads. go.upgradejs.com/n77
AI discovery starts with interviews, not tools. You cannot automate a workflow nobody described. www.ombulabs.ai/ai-strategy-...
Traced your LLM app? The same spans that debug a bad run are the raw material for an evaluation system. We just pointed our tracing post at the follow-up. go.upgradejs.com/3zn #LLMOps
A model reviews our time entries and surfaces only the ones worth a second look. That split is the design work. www.ombulabs.ai/case-studies...
Not everything deserves a span. LLM calls, tool calls, retrieval, prompt version: yes. Generic infra spans: no, they bury the four things you actually need. go.upgradejs.com/don #LLMOps
Teams running agents in production do not pick one top-ranked model. They route different tasks to different models, and they stop upgrading once task completion stops moving. go.upgradejs.com/v21 #AIAgents
Classify and drop non-events early and the expensive part of an AI pipeline never sees them. Some errors still slip through, which is why a human reviews the flagged ones. go.upgradejs.com/3ac #DataEngineering
You do not have to hand-roll spans. One package plus an OTLP endpoint gets a full LLM trace into Phoenix or Langfuse without touching your agent code. go.upgradejs.com/0aq #LLMOps
30 to 40% of LLM-generated selectors fail on the first try. Blind retries do not help. Classify the failure (too narrow, too broad, wrong region) and cap it at 3 attempts before a human looks. go.upgradejs.com/z0w #WebScraping
An assessment is only useful if someone builds what it recommends. Ours ends with a roadmap, a security framework, and config files your team owns, ready to hand straight to implementation. go.upgradejs.com/9qc #AI
Sometimes the honest answer is that a script, a form change, or a better report beats a model. Deciding not to build the AI feature is a real outcome, not a failure. go.upgradejs.com/dlv #AIStrategy
The control surface for a ReAct agent is the system prompt and the tool schema. Ours refuses to save until the post has both a title and an image, and the tool rejects anything under 20 characters. go.upgradejs.com/xhd #AIAgents
Not every entry needs a model. Hard rules handle the common cases, a binary classifier takes the ambiguous ones, and misfiled billable time dropped 95%. go.upgradejs.com/69p #MachineLearning
