The datatoad book (www.frankmcsherry.org/datatoad/cha...) now comes with a WASM playground to try out some of the problematic Datalog it handles. The book is not fully revved to always have runnable examples, but I'll get on that!

I wrote about "directional predicates" in datatoad. Treating predicates as relations can be much more powerful than predicates as .. predicates or even functions. Computation can go faster, because you reveal more to the computer about how to get work done. github.com/frankmcsherr...

I got Claude to help with a blog index refresh, surfacing about 1.5 years of posts to the front page that had been languishing: github.com/frankmcsherr.... It unintentionally included one unfinished post that I'll now have to get sorted out. :D

GitHub - frankmcsherry/blog: Some notes on things I find interesting and important.

Some notes on things I find interesting and important. - frankmcsherry/blog

github.com

Made a thing: Pollard is a MCP server for analyzing Firefox performance profiles. crates.io/crates/pollard

crates.io: Rust Package Registry

crates.io serves as a central registry for sharing crates, which are packages or libraries written in Rust that you can use to enhance your projects

crates.io

We now have datatoad's mdbook section on joins. It covers the planning of joins of abstract directed relations, and their execution by a mix of binary and worst-case optimal join stages. It's pretty tight, but I don't think it's missing much except examples. www.frankmcsherry.org/datatoad/cha...

As part of an attempt to get Claude to help me communicate better, we have .. a Claude-assisted mdbook for datatoad: www.frankmcsherry.org/datatoad/int... Still a work in progress, but it came together much faster with help.

Totally innocent question: I've found Claude has made me rather productive, at a rate that outpaces what I can write about solo. I'm trying to figure out if there is value/space for an intentionally "slop" line of posts that explains work that has been done, minus the alleged charm and certain salt.

A very nice post out of @materialize.com about "representation types" which remove non-structural distinctions between types as they approach the dataflow rendering layer, unlocking physical optimizations across logically distinct types: materialize.com/blog/no-clas...

No Classification without Represention | Materialize

Learn how Materialize improves query performance by compiling SQL's type system to a simpler system of

materialize.com

Fun experiment du jour: can Claude write stable matching in differential dataflow using a simplified language that hides any of details of DD. Yes, it turns out! Correct out of the box, and with a bit of nudging it got something better than I have written:

With Claude, and a few lines of Rust, you can add a live timely dataflow visualizer to your programs, giving you visibility into where the time goes, and soon (surely) arrangement record counts as well!

A new blog post about accelerating timely dataflow by 100x. Part of the story is how timely is able to do this, and .. seemingly .. other stream processor are not. It comes down to a fundamentally different approach to tracking time: globally rather than locally. github.com/frankmcsherr...

I used Claude for the first time this month, having only ever used copilot before (and .. stopped immediately afterwards). It turns out .. I am maybe obsolete? We improved `columnar` together, and each wrote a post about it. (linked within my post).

A recent post by @dov.dev about Materialize's new Iceberg sink, and generally about connecting live operational data to analytics stores. Some interesting detail about (overcome) streaming-batch friction, and turning pristine CDC streams into .. Iceberg! materialize.com/blog/making-...

Making Iceberg Work for Operational Data | Materialize

Apache Iceberg was built for batch analytics — but operational data changes continuously. Learn how Materialize streams live, transactionally consistent data into Iceberg without the memory and latenc...

materialize.com

Worlds collide: pretty sure someone at the MTA plays FFXIV:

I wrote a tiny vectorized interpreter last night, and .. thought I'd write up a post describing it. Nothing earth shattering here, but if you haven't seen one of these before it could be interesting! Probably easier to read than the J Incunabulum, but also less .. wow. github.com/frankmcsherr...

We have a recent @materialize.com post from Jan Teske about how MZ's self-correcting materialized views work. It's imo a very cool thing that unpacks some of the magic about how this could possibly work as the underlying system evolves. materialize.com/blog/self-co...

Self-Correcting Materialized Views | Materialize

Learn how Materialize uses self-correction to prevent output drift in materialized views, ensure consistency across upgrades, and enable in-place view replacement.

materialize.com

I wrote a bit about the "demand transform" for Datalog (and other similar languages): github.com/frankmcsherr... The idea of the transform is that rather than eagerly produce all the data you might ever need, you can start from the seed crystals of concrete values and data that might yield output.

If anyone ever told you Datalog is elegant, they probably haven't used datatoad yet. The evenness condition is an interesting example of relation-rather-than-function, though!

I went through and validated the output counts (numbers of facts) for most of the datatoad outputs. Some were hard to validate because datatoad sneaked in on my laptop but the other systems paged too hard, but everything seems correct. github.com/frankmcsherr...

Frank McSherry@frankmcsherry.bsky.social · 8mo ago

I wrote a post evaluating datatoad using the framework from the recent FlowLog paper: github.com/frankmcsherr.... It turns out it does well in some cases, worse in others, and has already improved by having other folks shine a light on its limitations by choosing problems and datasets I ignored!

I put up a new post about work that I am perhaps irrationally excited about: the combination of worst-case optimal joins with relational programming. github.com/frankmcsherr... The high-order bits are that relational programming is very neat, and it fits great with modern (WCO) relational joins.

I wrote a post evaluating datatoad using the framework from the recent FlowLog paper: github.com/frankmcsherr.... It turns out it does well in some cases, worse in others, and has already improved by having other folks shine a light on its limitations by choosing problems and datasets I ignored!

I wrote about datatoad as a "worst-case optimal Datalog", which .. perhaps doesn't typecheck. A previous result on streaming worst-case optimal joins seems to connect, and swapping "streaming" for "iterative" gives something that may be syntactically correct catnip. github.com/frankmcsherr...

Getting crisper; now within ~1s (14.8s v 13.7s) of my hand-optimized plan that uses only binary joins, but without any plan hints. Erm, mostly. Hoping to bypass it, as a win for WCO joins. :D

Frank McSherry@frankmcsherry.bsky.social · 9mo ago

I wrote a little bit about worst-case optimal joins in datatoad, support for which has just landed although there is a bit of tidying still to do before they are as crisp as the non-wco joins. github.com/frankmcsherr...

I wrote a little bit about worst-case optimal joins in datatoad, support for which has just landed although there is a bit of tidying still to do before they are as crisp as the non-wco joins. github.com/frankmcsherr...

A short note on how one can perform joins of sums by accumulating at a point halfway through the join. Neither the input nor the output of the join, but smaller intermediate terms that show up when you peer closer at what a join does internally. github.com/frankmcsherr...

I wrote a bit about datatoad going "full columnar". All operations are now columnar; no rows are ever formed; everything is column-at-a-time. github.com/frankmcsherr... A bunch of interesting (to me) algorithms, and also some performance regressions, but then clawing back. I learned things!

Looking forward to this!

Sync Conf@syncconf.bsky.social · 10mo ago

Featuring 👉 @notion.com 👉 @figma.com 👉 @swyx.io 👉 @threepointone.bsky.social 👉 @frankmcsherry.bsky.social 👉 @kyle.bricolage.io 👉 @jamescowling.dev 👉 @aaronboodman.com 👉 @schickling.dev 👉 @anselm.io 👉 @adamwiggins.bsky.social 👉 @powersync.com 👉 @b5.bsky.social 👉 @f0a.org 👉 @cdata.earth

New from Materialize: Cloud M.1 Clusters Run 3x larger workloads with the same low latency and predictable performance—thanks to intelligent data spilling and expanded capacity. Learn more: bit.ly/3L12oH2

Introducing New Materialize Cloud M.1 Clusters

Introducing a new Materialize Cloud cluster type. M.1 Clusters provide customers with more capacity, leading to better economics and performance, while maintaining the same low latency requirements th...

bit.ly

Datatoad check-in: this time including some recent progress on columnar joins (good news: faster). Though, it also tries to roll up a bit of the sprawl of content I've scribbled, which increasingly feels like it needs some more careful curation to be helpful. github.com/frankmcsherr...

Good news on the Datalog front: v1 of "columnar joins" seem to work, and resulted in a 20% improvement (from 9.5s to 7.5s, for the joins of a reference workload). Still more gains from tightening it up, and potentially from columnar sorting, but I'll take a swing at writing things up tomorrow!