Fred Hebert

@ferd.ca

Principal SRE @ honeycomb.io, Tech Book Author, Resilience in Software Foundation board member, Erlang Ecosystem Foundation co-founder, Resilience Engineering fan. SRE-not-sorry. blog: https://ferd.ca notes: https://ferd.ca/notes/

I think it’s now been more than a year since I last wrote software just for fun (with or without AI). It feels more like chores or work by now. It feels like it had been a long time coming, but it just became clearer recently.

I just realized I hadn’t shared this one yet, but I wrote about how the team I’m on restructured its workflows to lean into the code review bottleneck rather than trying to eliminate it when code generation took over, and what that ended up doing: www.honeycomb.io/blog/embraci...

How I Came to Embrace the Code Review Bottleneck

Faced with an endless stream of AI-generated code reviews, our team made the counterintuitive choice to lean into the bottleneck rather than reduce it.

honeycomb.io

Fix code review bottlenecks by doing like private torrent trackers and only allowing people to get their PR reviewed if they reviewed enough PRs beforehand to keep their ratio high enough.

The first rule of the papers-reading club is that you’re unfortunately already in the papers-reading club and I will send you links and summaries

Yet again hearing how some skilled work never truly mattered now that it's getting automated, while it absolutely did and was a point of professional pride for many. It's a significant aspect of 'deskilling', which is a known consequence of automation. Pretending otherwise is needlessly unkind.

Although there have been changes in cognition and systems theory, I wanted to bring up some stuff Rasmussen published in the 80s that was really elegant. I'll use 3 diagrams he published, covering the ideas of abstraction hierarchies and how people operate systems when troubleshooting them.

Wrote up a bunch of stuff about some patterns in system design, about the tension, contrast, and possibility of composing approaches of analytical decomposition to increase control, and of complexity-aware stances for emergence, and some pitfalls of either stance: ferd.ca/control-and-...

Control and complexity: tension in systems design

composing two broad approaches, one based on analytical decomposition that aims to maintain control over a system, and one based on a perspective of complex systems that resist analysis, and implicati...

ferd.ca

Delegating pressure to the final individual, who is now tasked with continuously fixing the entire system’s misalignments through their personal choices.

One of the ironies about AI agents in ops tasks is that it feels like there has never been as much interest in creating a forgiving environment with proper structural support than through promising to remove humans from it, finally forcing a less individualistic and blameful approach to design.

Even after years, one of the weirdest parts of gardening to me is needing to harden the seedlings before transplanting them. Like “yes hold on a minute I gotta take the plants out so they can play outdoors for a while, but they gotta be in before streetlights turn on” is a real and necessary thing.

The ongoing stream of software engineering pieces that mention that the future is in writing spec but never bother to define what a specification is or at what abstraction levels it should be is appalling; arguably, tickets are a spec, the code is a spec, and work between both is connecting dots.

“[…] we reached for Recon, an amazing tool for diagnosing issues […] (the related Erlang in Anger is more-or-less required reading as all on-call engineers end up scouring its pages eventually)” Wild! I wrote these 10 years ago to help coworkers, and they’re still useful for real world issues now!

You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage

On March 25th, voice and video on Discord suffered major degradation beginning at 12:13 PDT, lasting a little over three hours. Learn how the issue originated, how it affected systems across Discord, ...

discord.com

Infinite love to my SRE coworkers, one of whom casually dropped this single line in a retro: "the industrial hourly deploy train and its consequences have been a disaster for society"

I wrote for @resilienceinsoftware.org on "Superficial Blamelessness", where under the label of "blamelessness", we avoid punishing people, yet still focus fixes and interventions based on the same individualistic framing rather than a broader systemic stance. resilienceinsoftware.org/news/11502437

Superficial Blamelessness

In 2012, John Allspaw (then CTO of Etsy) wrote a seminal blog post on the need for what he called Blameless Postmortems. Built off the notion of a “Just Culture” from the research of Sidney Dekker, he...

resilienceinsoftware.org

I've gotten an early copy of Crisis Engineering by Marina Nitze, Matthew Weaver, and Mikey Dickerson, and just in time for its release today, here's my review of it: ferd.ca/notes/on-cri... TL:DR; I like it, good perspectives and an interesting mix of approaches.

One of my assignments asks whether a given set of psychological constructs amount to “good science” and it’s very weird as someone who’s written zero papers and done no real science to just attempt to go “sure dawg it’s okay science I guess but these hundreds of researchers could pick better models”

Every time I am faced with this Swedish login form, I have to say 'Logga in' out loud with a terminator voice. It is one of the few rules we can't change and must be respected.

swedish login from with a button that says 'Logga in'

hell yeah @resilienceinsoftware.org swag is in! the law of requisite variety states that only variety in the regulator can destroy variety in the system being regulated. so if you need to deal with complexity you know you gotta join the club & begrudgingly increase complexity to keep things simple

Black hoodie showing 'anti complexity complexity club' in a red explosion based on the logo of the Resilience in Software Foundation.