Papers We Love is back this Thursday in downtown SF! I'm presenting "Simple Testing Can Prevent Most Critical Failures," an empirical evaluation of real distributed systems bugs with surprising conclusions. Come for the banh mi and drinks, stay for the conversation. luma.com/zorjnr9f

I'd be so much more aggressive with AI for my email if Google didn't couple creating drafts and sending them into one "gmail.compose" OAuth scope. Some PM in Google Apps must be desperately searching for an AI project before the next promo season - be the hero we all need!

Straight 90s vibes for me today: editing today's mega-doc in the Terminus font, lots of Eve, Nas, Dre, and Outkast in the playlist, and a too-large AWK script on the side.

Nothing like running face-first into a sentence including “prebendaries enfeoffed in kind or in money” to emphasize how long it’s been since I’ve read something fancy.

Don't miss Papers We Love Too this week! On Thursday (Feb 19), join us in downtown SF to hear Marco Primi present "Paxos Made Simple" - the most famous algorithm in distributed systems. RSVP on Meetup (www.meetup.com/papers-we-lo...) or just DM me your full name and email.

@paperswelove.org is back: on Feb 19, join us in downtown SF to read "Paxos Made Simple" with Marco Primi! We'll have banh mi, drinks, good papers, and good conversation. RSVP on Meetup (www.meetup.com/papers-we-lo...) or just DM me your full name and email.

Ain’t no city like my city.

After a few quiet, online-only years, @paperswelove.org is back in San Francisco! Our first in-person meetup is next Thu, Jan 22: @shadaj.me on "Perceus: Garbage Free Reference Counting with Reuse" and me on "Epidemic Algorithms for Replicated Database Maintenance."

Come reboot Papers We Love in San Francisco! Shadaj Laddad and I will be kicking things off on Jan 22 with papers on garbage collection and gossip protocols.

Papers We Love@paperswelove.org · 7mo ago

Akshay Shah on "Epidemic algorithms in replicated databases" - January 22nd in San Francisco 📜 Paper: dl.acm.org/doi/pdf/10.1... 📆 RSVP: www.meetup.com/papers-we-lo... #sfo #meetup #paperswelove #databases #research #programming

Property-based testing's roots in academia and functional programming give it a bad rap. It's easy to get started, and it's a really high-octane investment for most projects.

Internally, AWS implements consensus as a platform primitive, which eventually evolved into a distributed, strongly consistent, transactional journal. It backs Kinesis, DynamoDB streams, & Aurora DSQL. I'm really surprised that there isn't a battle-tested OSS version.

Baffling how the same people can be enraged by 5m builds and enthralled by multi-day async decision-making processes.

Just in time for my recent re-obsession with gossip-based systems: Fly.io published a blog post on moving from multi-region Consul to Corrosion, their in-house, SWIM-based metadata store. fly.io/blog/corrosi...

Corrosion

Corrosion is distributed service discovery based on Rust, SQLite, and CRDTs.

fly.io

For everything today's models are bad at, I'm really happy with how the images in this presentation came out. Starting with our designer's Halloween images and a pic of myself, then riffing from there was fun! The model feels like an in-betweener in the hand-drawn cartoon days.

🧵 OK, so I just read Google's "Firefly: Scalable, Ultra-Accurate Clock Synchronization for Datacenters." Lots of cool ideas. But how do people *monitor* decentralized systems like this?

she is @antithesis.com

New mascot day! Here's @akshayshah.org introducing our new mascot to a packed audience at @leaddev.com NYC. We're very sorry about what happened to the old mascot.

"Popularized in Haskell" Denominators can really do a lot of work.

In honor of spooky month, share a 4 word horror story that only someone in your profession would understand. “Fifty-two bit mantissa.”

Kris 🧙‍♂️ Kowal@kriskowal.com · 10mo ago

In honor of spooky month, share a 4 word horror story that only someone in your profession would understand. "Off by one"

Datadog must have been polishing this piece of ragebait for years. It's art, really. (But seriously, this API is like a cheese grater to the face.)

Vibe-writing in Excel? No thanks 🤮 Unreliable tools can be incredibly useful - but you need a bulletproof test for their output. And spreadsheets are famously hard to get right! 94% of operational spreadsheets contain errors, and humans are exceptionally bad at catching them.

Best convo on the internet right now @vicvijayakumar.com

How AWS S3 serves 1 petabyte per second on top of slow HDDs // Stanislav Kozlovski Obscure feature + obscure feature + obscure feature = bug // Michael Gibson, @antithesis.com %CPU utilization is a lie // @brendanlong.bsky.social

This is *so* clever! At least in my head, this rhymes with the MapReduce philosophy of moving compute to the data.

Andy Pavlo@andypavlo.bsky.social · 10mo ago

Our F3 files embed small WASM programs to decode data. If somebody creates a new encoding and the DBMS does not have native impl, it can still read data using WASM passing Arrow buffers. Our experiments show WASM is 15-20% slower than native. We use @spiraldb.com's Vortex encoding impls.

I’d rather write statsd to the wire by hand than descend into the Lovecraftian horror of the OTel Go docs.

Unexpected benefit of saving PDFs of all the papers and @oreilly.bsky.social books I’ve read over the years: I have the most awesome NotebookLM session for distributed systems.

Find out why AI code gen delivers only partial wins today. Join us on Oct 9 for a live webinar with @antithesis.com's Will Wilson & @akshayshah.org. They'll share how to bridge the gap between AI generation and real engineering productivity. Don't miss it ➡️ bit.ly/Antithesis-A...

HBR is usually hot garbage, but you gotta respect whoever came up with “workslop” 💯 hbr.org/2025/09/ai-g...

AI-Generated “Workslop” Is Destroying Productivity

Despite a surge in generative AI use across workplaces, most companies are seeing little measurable ROI. One possible reason is because AI tools are being used to produce “workslop”—content that appea...

hbr.org

“Antithesis’s deterministic simulation and active exploration remove the blindfold, enabling a systematic and reproducible search for bugs.” - Marek Siarkowicz, Kubernetes maintainer www.cncf.io/blog/2025/09...

Autonomous Testing of etcd’s Robustness

As a critical component of many production systems, including Kubernetes, the etcd project’s first priority is reliability. Ensuring consistency and data safety requires our project contributors to…

cncf.io