this post is currently happening eastside...may their cold pizza rot fast and their regret sting ever fiercer RE: https://infosec.exchange/users/k3ym0/statuses/117607080678996785
k3ym𖺀 (@k3ym0@infosec.exchange)
INT. REDMOND, BUILDING 34. 3:47 AM. A conference room that smells like cold pizza and regret. Office 365 EVP: Talk to me. Why is Exchange still down? MID-LEVEL MANAGER (Dave): So, good news and bad news. EVP: Give me the good news. DAVE: We found the service that's failing. EVP: Great. Who owns it? DAVE: That's the bad news. It's called auth-bridge-final-v2-REAL. The commit history says it was written by "Copilot" and approved by "Copilot." EVP: Then get the engineer who prompted it. DAVE: That was Priya. She was in the Q2 "efficiency realignment." EVP: Her backup? DAVE: Also realigned. So was her backup's backup. The whole team is a Slack channel now, and the only one in it is a bot that posts "🚀 LGTM" on every PR. EVP: Fine. Then we read the code and fix it. DAVE: We tried. It's 340,000 lines of Python that calls a Rust binary that calls a PowerShell script that curls a Gist. The Gist's README just says "dont touch this lol." EVP: Documentation? DAVE: One Confluence page called "Architecture (TODO)." Last edited in 2024. The only content is a Mermaid diagram that fails to render. EVP: Jesus Christ. Can we roll back? DAVE: To what? The rollback pipeline was also vibe coded. The last time someone ran it, it rolled back to a version that only existed in the AI's hallucination of what the service should look like. EVP: So what the fuck have we actually been doing for nine hours? DAVE: Rebooting servers one at a time and praying. Comms is calling that "restarting affected infrastructure." EVP: ...Is there anyone left who understands how any of this works? DAVE: Gary. EVP: Who's Gary? DAVE: Contractor. 61 years old. He wrote the original Exchange routing logic in 2003 and he's the only one who knows why there's a sleep(7) in the auth handshake. We let him go in January, but HR forgot to disable his VPN. EVP: Call Gary. DAVE: Gary says his rate is now $900 an hour, with a 40-hour minimum, and he wants it in writing that he was right about everything. EVP: ...Pay Gary. DAVE: Also, the board wants to know if this will affect the AI efficiency numbers for the earnings call. EVP: Tell them the outage was caused by "an unexpected interaction during a routine change," and that AI agents are "actively assisting with remediation." DAVE: Are they? EVP: Copilot opened a ticket. DAVE: What did the ticket say? EVP: "Have you tried turning it off and on again?"
infosec.exchange