A deep dive to how metrics are stored and queried. Inverted indexes, roaring bitmaps, and ... gorillas? www.bitsxpages.com/p/how-metric...

how metrics are stored and queried

the data structures powering your metrics dashboard

bitsxpages.com

The team worked insane hours to get this release ready and we've finally shipped it! We built a first of its kind, prometheus-compatible database that is object native. ~22x cheaper than AWS AMP and stupid easy to operate. www.opendata.dev/blog/introdu...

OpenData Timeseries: Prometheus-compatible metrics on object storage | OpenData

An MIT-licensed, Prometheus-compatible timeseries database built on SlateDB, bringing the operating model and cost structure of object-store-native systems to the Prometheus and Grafana ecosystem.

opendata.dev

I know "measure twice, cut once" applies to performance optimizations, but where's the fun in that? I am quite proud of my lopsided planter boxes and custom memory allocators.

Databases have strange economic incentives that end up making them "worse" over time. The problem is an asymmetry between who benefits from innovations at first (database companies) and who benefits in the long run (hyperscalers that sell hardware). Blogged: www.bitsxpages.com/p/the-broken...

the broken economics of databases

Why database companies charge so much, earn so little, and keep making things complicated

bitsxpages.com

I regularly get asked "how do you create your diagrams for your blogs"? Recently I've been using my own tool, and I'm open sourcing it! Yuzudraw is a visual editor for ASCII that has a token-efficient DSL that lets agents generate/modify your diagrams. Give it a ⭐! github.com/agavra/yuzud...

GitHub - agavra/yuzudraw: ASCII diagramming tool for both humans and agents

ASCII diagramming tool for both humans and agents. Contribute to agavra/yuzudraw development by creating an account on GitHub.

github.com

“Gradually, then suddenly.” That’s how adoption works when you’re building something new. Opendata is still "gradually" but 100 stars with $0 spent on marketing is a good start. Back to building! github.com/opendata-oss...

I implemented prefix compression for SlateDB & noticed benchmarks looked "worse". Fell down a rabbit hole. Turns out I was thinking about compression backwards... Wrote up my learning: www.bitsxpages.com/p/the-mathem...

the mathematics of compression in database systems

why compression is (almost) always worthwhile

bitsxpages.com

Can you beat 180KB? I created a challenge to reduce a dataset as much as possible 🏆 my approach uses delta encoding & prefix compression + zstd(22) compression to reduce 25MB -> 180KB github.com/agavra/bit-g...

GitHub - agavra/bit-golf: a compression golf challenge for GitHub event data

a compression golf challenge for GitHub event data - agavra/bit-golf

github.com

I fixed the problem with reviewing code written by Claude in "accept edits on" mode: github.com/agavra/tuicr would love to know what you think, and if you want to contribute to an OSS rust project there's a bunch of open issues to pick up!

I'll die on this hill: Sorted String Tables (SSTs) are the single most important data structure for modern DBs. They lean in to the limitations of SSDs and Object Storage, making them (and similar layouts) the best choice for many databases. Blogged in detail: www.bitsxpages.com/p/sorted-str...

sorted string tables (SST) from first principles

why sorted string tables are the swiss army knife for data systems and how they are implemented

bitsxpages.com

The inner join between sets of people who build databases, write, and draw? Low cardinality. I'm in that set, so I'm starting a blog! Here's my first post: www.bitsxpages.com/p/frameworks...

frameworks for understanding databases

building mental models for tradeoffs in performance, availability and durability in data systems

bitsxpages.com

Sometimes the best solution is "do nothing", but it's always more fun to play with tools.

Calling database nerds in SF! I'm covering SlateDB at the systems meetup next Wednesday (10/29). If you're around, I'd love to meet you in person (that way you'll have proof I'm not just an AI bot). 👉 luma.com/e7feg2i6

Beans on toast is underrated.

I recently implemented Gorilla encoding (www.vldb.org/pvldb/vol8/p...) for a SlateDB PR. Pretty cool stuff - easy to understand but really powerful. Here it is, explained by a gorilla.

It's true. Every company eventually becomes a database company.

Despite using so many new technologies, I somehow never learn my lesson: read the docs sooner and read the docs thoroughly.

Reading the tokio-rs async documentation makes me feel like...

One day I'll open a coffee shop dedicated to the not-insignificant intersection between database nerds and coffee snobs. Until then, enjoy this comic.

Maybe... just maybe, adding more features and complexity into stream processors is NOT what we need?

The new electric Caltrain cars have WiFi. 🙏 SF bay area has finally entered the 21st century (on this dimension of public transit only).

Today marks SlateDB’s one year anniversary! It’s been a lot of fun. Thanks to @rohanpd.bsky.social @flaneur2024.bsky.social @almog.ai @vigneshc.bsky.social @paulbutler.org Jason Gustafson, David Moravek, and many others for joining the project. 😀

SlateDB - An embedded storage engine built on object storage | SlateDB

Description will go into a meta tag in <head />

slatedb.io

Prompt engineering was v0. Context engineering is v1.0. How should you think about supplying LLMs with the right information? 📖 [1/N]

"Kafka Configs: A Rube Goldberg Machine?" - an actual quote from a customer after discussing cleanup.policy and Kafka Streams changelogs.

SlateDB now has clones 🤯 Users can clone an existing DB's data to a new location. It's nearly instantaneous since it references the data from the old bucket rather than copying. Writes to the clone update the new location. Compaction lazily merges old data into the new directory./ht @responsive.dev

Initial clone implementation by hachikuji · Pull Request #430 · slatedb/slatedb

Fixes #315. This patch contains the logic to create and initialize clones as outlined in RFC-0004.

github.com

Should you use Kafka? If so, when? And what are the tradeoffs presented by the dizzying variety of Kafka-adjacent technologies? I hope that my latest blog post provides unique and useful answers to these questions. Let me know what you all think! 👉 www.responsive.dev/blog/why-whe...

As part of my post-Elden Ring hunt for a game, I tried Balatro last night. I'm starting to think I may have made a mistake.

How did Responsive simplify its architecture and achieve monstrously high availability with unlimited scale potential? Co-founder @almog.ai will share how replacing RocksDB with #ScyllaDB in #Kafka Streams led to big wins for their team at Monster Scale Summit. www.scylladb.com/monster-scal...

My hobbies all have the ability to "reclaim" waste: 🪴 for ceramics, wasted clay from throwing goes right back in to your next pot 🪵 for wood working, reusing 2x4s is second nature 🥘 for cooking, leftover food is tossed into salads 💻 what's for software? bin-packing containers? 😆

Kafka Streams 101: Windows & Time! 🕰️ What's the difference between event, stream and wall clock time? 🪟 What are the four different types of windows? ⁉️ What are the important error messages and metrics, and what do they mean? 👉 Read the full lesson here: www.responsive.dev/blog/windows...