antithesis.com/blog/2026/fi...

Finding bugs in Raft implementations | Antithesis

Why formal verification is not enough to maintain consensus in real systems

antithesis.com

We've been testing Raft implementations, and have found bugs in every implementation we've tested. Here's an in-depth report on one set of issues.

Great recent find “Epic: The Musical”., based on The Odyssey. Gods, monsters, magic, love, death, betrayal, revenge, drama. It has It all. **So quotable**. music.youtube.com/playlist?lis...

EPIC (FULL Official Concept Album)

Full Official Concept Album of EPIC with all songs in order! :D

music.youtube.com

#SD25 online continues today! Join Kyle Kingsbury for Jepsen 18: Serializable Mom How three unconventional systems ensure--or violate--key safety properties 10am PT / 1pm ET / 7pm CET www.youtube.com/watch?v=dpTx... @jepsen.mastodon.jepsen.io.ap.brid.gy

Jepsen 18: Serializable Mom by Kyle Kingsbury

YouTube video by TigerBeetle

youtube.com

------- "Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25 Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases. "I professionally set those tires on fire".

It's day two of Systems Distributed, hosted by @tigerbeetle.com! I'll be liveskeeting all of the talks, except mine (at 11 AM). Since the venue is a film museum, they're setting up special posters for each talk. #sd25

Greetings from #sd25! It's a two-day single-track conference hosted by @tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here! systemsdistributed.com

Systems Distributed '25

A conference to teach systems programming and thinking, and how to apply these ideas. All the way across the stack. From systems languages and compilers, to databases and distributed systems.

systemsdistributed.com

A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]

A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]

All the videos are up at www.hytradboi.com/2025#program now.

"Correctness at Feldera" talks about various correctness techniques, including: - machine proof of the underlying DBSP algorithm - differential testing of the implementation https://www.feldera.com/blog/correctness-at-feldera

Correctness at Feldera

In this blog post, we briefly describe our efforts and development processes that ensure Feldera's engine is correct.

feldera.com

Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...

Databases in 2024: A Year in Review

Andy rises from the ashes of his dead startup and discusses what happened in 2024 in the database game.

cs.cmu.edu

Curated list of materials on testing SQL database engines is public now github.com/asatarin/tes...

GitHub - asatarin/testing-sql-databases: Curated list of materials on testing SQL database engines

Curated list of materials on testing SQL database engines - asatarin/testing-sql-databases

github.com

This is a sample from a list "testing-sql-databases" I have. It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.

Good old Microsoft published some work: - "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1... - This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0

Deploying a Steered Query Optimizer in Production at Microsoft | Proceedings of the 2022 International Conference on Management of Data

You will be notified whenever a record that you have chosen has been cited.

dl.acm.org

Work on correctness of optimizers from Greenplum - "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/... - "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...

Automatic capture of minimal, portable, and executable bug repros using AMPERe | Proceedings of the Fifth International Workshop on Testing Database Systems

You will be notified whenever a record that you have chosen has been cited.

dl.acm.org

Similar work from Databricks: - "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1... - "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI

You already mentioned "Snowtrail: Testing with Production Queries on a Cloud Database" from Snowflake dl.acm.org/doi/10.1145/...

More correctness work from MongoDB, not yet on my list "Verifying Transactional Consistency of MongoDB" arxiv.org/abs/2111.14946

Verifying Transactional Consistency of MongoDB

MongoDB is a popular general-purpose, document-oriented, distributed NoSQL database. It supports transactions in three different deployments: single-document transactions utilizing the WiredTiger stor...

arxiv.org

MongoDB team did great work on performance: - "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/... - "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1... - "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...

Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings of the ACM/SPEC International Conference on Performance Engineering

dl.acm.org

To counter balance your argument some big tech work on correctness of (SQL) database in a thread below. Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface: - "Randomized Testing of Cloud Spanner" medium.com/@jcorbett_26...

Get your point, but to be fair, this list is not targeting correctness of databases (SQL or otherwise). It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in asatarin.github.io/testing-dist...

Testing Distributed Systems

Curated list of resources on testing distributed systems

asatarin.github.io

I'd be happy to learn that this is just a gap in my testing knowledge, or that there's a bunch of secret testing systems in big companies that I don't know about, but even just skimming the list of asatarin.github.io/testing-dist... kind of illustrates my point. (Thanks again @asatarin.bsky.social!)

Testing Distributed Systems

Curated list of resources on testing distributed systems

asatarin.github.io

Deterministic simulation has been entirely pushed by startups. Jepsen is jepsen. All the SQL fuzzing stuff I know of comes from academia or database startups. Larger companies seem to only be winning in the application of formal methods (because they can afford to hire a team of ex-professors?)

When I think of advancements in quality and correctness of databases, it feels most things have come from startups or individuals, and not large well-established companies. Which seems... backwards? We talk of startups hacking out code and megacorps crawling to keep a high quality bar.

Alexey's @clickhouse.com talk here is hilariously off-the-chain. I highly recommend watching it: www.youtube.com/watch?v=jmVx... You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.

My Database Can Do This by Alexey Milovidov (DBDBD 2024)

YouTube video by DSDSD - Dutch Seminar on Data Systems Design

youtube.com

MariaDB is improving their isolation level guarantees following the recent @jepsen.mastodon.jepsen.io.ap.brid.gy report. "Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.

Isolation level violation testing and debugging in MariaDB | MariaDB

Explore how the InnoDB team tackles isolation violations in databases, highlighting debugging strategies and a new option for Snapshot isolation compliance.

mariadb.com

Alexey Gotsman talks about database isolation levels. Starts off with consistency in general: - what is strong consistency - common misunderstandings and pitfalls - serializability and linearizability - weak isolation levels - safety and liveness Video links below.

Hi, I’m a distributed systems researcher at MongoDB, photographer, Zen Buddhist in the White Plum lineage, rock climber living in New Paltz NY. I hope to meet fellow researchers here, and I hope we can all settle on BlueSky as our Schelling point for a couple years before we have to do this again.

Hello, Bluesky users! I curate and maintain list of resources on testing distributed systems. You might have seen it before. It's a good one, if I may say so myself. asatarin.github.io/testing-dist...

Testing Distributed Systems

Curated list of resources on testing distributed systems

asatarin.github.io