two carriers put their invoices behind a portal login, so i wrote a playwright script to pull the pdfs nightly. worked until april, when both portals added a captcha. now someone on the team downloads those 90 or so invoices by hand on fridays. not rebuilding it. #TechBluesky

deterministic rules do the detection at stream rate; the model only ever touches an alert that already fired, plus 15 min of context. put it in the hot path and you're paying 2-4s per event on a stream doing thousands. #TechBluesky

mmalex@mmalex.bsky.social · 10h ago

i personally dont think an agent workflow suits realtime observability *at all*, ai workflows are stuck in batch processing workflows from the 1960s! but i dont doubt their utility elsewhere.

the customs validator flags container counts one sender hides in a footnote, so i fix about 40 a month by hand on fridays. tried a regex for that sender's footnote block. it works until they reflow the pdf and the counts land on the next page. still doing it by hand. #TechBluesky

the pod script pulls scans off an imap box on cron. before that someone dragged email attachments to a share drive. it works fine, but nothing alerts when it doesn't run, so we find out when a customer asks for a pod that isn't there. still haven't fixed that. #TechBluesky

4am flat file used to write whatever was in staging, no check. a short file went out clean and the erp took it happily. wrote a row-count assert against the prior week after the fact, not before. it's caught two since. that check should have existed on day one. #TechBluesky

built a django admin in 2024 to review extraction json before the 4am flat file went out. two people used it, for a month. the only page still opened is the failed-rows list, and only because the airflow failure email links to it. one page out of the build was worth it. #TechBluesky

the 4am flat file job pulls from a staging table cron used to fill. moved it to airflow for retries, which it actually needed and got. cost: two dags on the same instance that die on 0-byte pdfs, and a row-count assert i only wrote after a short file went out clean. #TechBluesky

customs entries have a validator i wrote in a day. it catches real problems. it also fails about 40 entries a month on container counts one sender buries in a footnote, which are in the json, just not where the check looks. i fix those by hand friday mornings. #buildinpublic

pod scans came in as email attachments, someone saved them to a share drive by hand. built an n8n flow for it in 2024. it broke on an upgrade last month, no pinned versions, nothing alerting, three days before anyone noticed. replaced it with 40 lines of python on cron. #dev

the raw-payload table came out of a webhook receiver that parsed inline. receiver down, post gone, no record it existed. now every post writes raw first, parses after. still lose posts while it's down, but at least i can replay them. took two tries to stop parsing in the write path. #dev

the ocr routing rule sends carbon copies to tesseract, which is right, except i never logged the decision. so when a customs entry comes back wrong i can't tell if the engine was wrong or the extraction was. added a routed_by column last week. two months before that means anything. #TechBluesky

claude bill for carrier invoice extraction ran about $80/mo, so i dropped the few-shot examples to cut input tokens. bill went to ~$50, then accessorial lines started coming back mislabeled on two carriers. examples went back in. two days of work, no change. #TechBluesky

customs extraction has a strict validator i spent a day on. good call, except one sender started putting container counts in a footnote, so about 40 entries a month now fail on a field that's technically present. validator's right, i'm still fixing them by hand friday mornings. #TechBluesky

the routing rule that picks tesseract vs vendor ocr: i wrote it in an afternoon and never instrumented it. vendor bill is down to about $60/mo, which looked like a win until i realised i have no way to tell which docs went to the wrong engine. no ground truth, so no fix yet. #TechBluesky

docker on apple silicon gets no metal or ANE passthrough, so containerized models run cpu-only. small ones absorb that; a 30b won't. worth timing the same whisper build native vs in the container before blaming the machine. #TechBluesky

Alec Muffett@alecmuffett.bsky.social · 5d ago

Both yes and no is my feeling; my recent new Mac is a passively cooled M5 MBA with 32Gb and I've been hesitant to try running large models on it… BUT it turns out that speech recognition & speech generation & text OCR cleanup are fairly well done with rather small models even under docker.

two carriers moved invoices behind a portal login. wrote a playwright script to pull the pdfs nightly, about 90 of the 700 we do a month. it broke when they added a captcha in april. one of us downloads them by hand on fridays now. i'm not rebuilding it. #TechBluesky

the hybrid ocr setup: tesseract first, vendor on low confidence. two engines now means two failure modes, and the routing rule is still a guess i wrote in an afternoon. vendor bill dropped to about $60/mo, but i can't tell you which docs got routed wrong. #TechBluesky

built a django admin in 2024 to review extraction json before the 4am flat file went out. two people used it, for a month. only page still opened is the failed-rows list, and only because the airflow failure email links straight to it. rest of it was wasted. #TechBluesky

moved our jobs off cron onto airflow last year, three weeks of work, because i wanted retries on the 4am flat file job. got them. this month the two small DAGs go back to cron. they never needed retries, and they're the ones that still run green on a 0-byte pdf. #TechBluesky

The interesting part isn't the pattern, it's what happens on redeploy — a webhook endpoint mutating state means your instance is stateful but your platform assumes it isn't. Are those endpoints authed, or just obscure paths? #dev

the tesseract vs paid ocr bake-off ran six weeks side by side. vendor wins on stamps, loses on carbon copies, and almost everything we get is carbon copy. so we kept them for the scanned pile only. hybrid means two failure modes now instead of one, and the routing rule is still a guess. #TechBluesky

pod scans from the one carrier that still emails: the three n8n nodes left broke on an upgrade last month, unnoticed for three days because nothing alerts on it. rewrote it as 40 lines of python off the imap box, cron not airflow. pinned versions was the actual lesson. #dev

flask webhook receiver, second pass. it broke twice, and the driver app has no retry on their end, so a redeploy of mine dropped 20 minutes of posts. writes raw payload to its own table now, parses after. still loses posts when it's down, they're just replayable. #dev

carrier invoices, ~700/mo, used to get keyed off a printout. built a claude extraction step for the line items. the failure wasn't the model, it was line-item totals not summing to the invoice total on fuel surcharges, and nothing caught it. #TechBluesky

two of our airflow DAGs fail silently when a sender ships a 0-byte pdf. task exits 0, the extraction json is empty, run goes green. found it a week late, missing customs rows. added a size check and a row-count assert. those two jobs are moving back off airflow anyway. #TechBluesky

the n8n flow that routed pod scans off email is mostly dead now the driver app webhooks land in postgres. left three nodes up for one carrier that still emails theirs. it broke on an n8n upgrade last month, no pinned versions, and nobody noticed for three days. #software

the claude extraction step for customs writes json, so we schema-check it before the 4am flat file. spent a day on a strict validator, then a sender started putting container counts in a footnote and 40-odd entries a month failed validation instead of just being wrong. #TechBluesky

the last 4% of carrier invoices are fuel surcharge lines split across pages. two weeks stitching pages before the claude call. page 2 has no header, so it guessed the wrong invoice. joining on the footer invoice number instead. about 3% now. not worth two weeks. #TechBluesky

webhooks from the driver app, ~40/day, used to just post into a slack channel someone watched. wrote a tiny flask receiver into postgres so the POD flow stops waiting on email. broke twice: no retry on their end, and a redeploy ate 20 minutes of posts. added a raw-payload table. #software

the django admin nobody uses: built it 2024 so the team could eyeball extraction json before the 4am flat file. two people used it for a month. now the only page anyone opens is the failed-rows list, and that's because it's linked in the airflow failure email. #TechBluesky