They’re calling it our most boring feature yet. boring(n): mundane; works exactly as expected; the highest praise you can give a database

turbopuffer@turbopuffer.bsky.social · last yr.

by popular demand, now introducing one-dimensional vectors 🎈

tpuf Python client went async today! 🐡 ❤️ 🐍 Async client perf slightly edges out sync client perf under heavy query load. github.com/turbopuffer/...

Big day today at turbopuffer HQ.

turbopuffer@turbopuffer.bsky.social · last yr.

turbopuffer is generally available. petabytes in prod. turbopuffer.com/

when we said "coming soon" we really meant it now puffin' in aws-us-east-1 and aws-eu-central-1

Things move fast at turbopuffer. Now puffin' in aws/us-east-1 and aws/eu-central-1 too.

I've been dreaming about conditional writes on S3 for years. I couldn't have asked for a better way to celebrate than getting to ship tpuf on AWS. 🐡

turbopuffer@turbopuffer.bsky.social · last yr.

now available: turbopuffer AWS regions ☁️ come find us in us-west-2 and ap-southeast-2

It’s a rare day that my love of going to battle with build systems pays off like this. Kudos to GCP for a very impressive new SKU. 🐡💨

turbopuffer@turbopuffer.bsky.social · last yr.

GCP released a new ARM machine type with NVMe SSDs recently 70% increase in end-to-end production indexing throughput at 20-33% lower machine cost!

Couldn't be more excited to be joining the team at @turbopuffer.bsky.social. 🐡💨

turbopuffer@turbopuffer.bsky.social · last yr.

welcoming two excellent additions to team tpuf this week: @arash11gt as a customer engineer, and @nikhilbenesch to the DB team from pasts at Materialize & Cockroach. we're locking in a p999 eng team to build the best search engine for scale.

Just catching up on my NULL BITMAPS and this is easily the best intuition for write skew I've ever seen described. The analogy to merge skew in a codebase is genius.

Justin@jaffray.bsky.social · 2y ago

this week on NULL BITMAP, intuition for snapshot isolation: it’s like git buttondown.com/jaffray/arch...

Well well well: www.crunchydata.com/blog/pg_incr... Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.

pg_incremental: Incremental Data Processing in Postgres | Crunchy Data Blog

We are excited to release a new open source extension called pg_incremental. pg_incremental works with pg_cron to do incremental batch processing for data aggregations, data transformations, or import...

crunchydata.com

Nikhil Benesch@benesch.bsky.social · 2y ago

Maybe EDB or Crunchy could pull it off, a la Crunchy Data Warehouse. But it’s not clear the business case is sound. IVM means more efficient read workloads, which means customers spend *less* money on Postgres.

The 6th (and last of 2024!) NYC Systems talks are next Thursday! We've got @jaronoff.com of Omlet and @benesch.bsky.social of Materialize speaking. :) nycsystems.xyz/december-202...

🚀 Postgres Compatibility Index (PCI): Think your shiny new Postgres derivative is "Postgres-compatible"? Test it, score it, and know the truth. 🐘 Thanks @gunnarmorling.dev @tudor.golubenco.org @benesch.bsky.social @deverts.bsky.social for the inspiration. Blog ==> tinyurl.com/mmfcbczz

PostgreSQL Compatibility Index: The Fellowship of the Database

In the mystical realm of databases, a new hero rises every few moons — a shiny, next-gen PostgreSQL derivative, boldly claiming to be…

tinyurl.com

DBA@pgdba.bsky.social · 2y ago

Something like this? Postgres compatibility index "alpha version" github.com/secp256k1-sh...

First new post in a couple of weeks! There's been a lot of activity around regattastorage.com this week, so I decided to write about the space. tl;dr It's pretty exciting!

The Quest for a Distributed POSIX-Compatible Filesystem

Distributed POSIX filesystems have proven elusive, but we're getting closer. Perhaps that's all we need.

materializedview.io

Today's third blog post on Aurora DSQL: this time looking at the write and commit path, and how conflicts are detected and handled: brooker.co.za/blog/2024/12...

For anyone keeping score in the open table format wars, AWS launching their s3 table bucket with only Iceberg support seems like a really big deal. Nothing is named in such a way that they can't add support for Delta down the line, but this feels like a huge endorsements of Iceberg (vs. others)

My second blog post on Aurora DSQL, with a deep dive on how reads work: brooker.co.za/blog/2024/12...

S3 (Iceberg) Tables is everything I dreamt of, and more. I blogged some long-form thoughts: meltware.com/2024/12/04/s... I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets. #databs

Aurora DSQL uses most of the SQL layer (parser, planner, optimizer, etc) from PostgreSQL itself. The storage layer, concurrency control, replication, etc are ground-up.

GCP is now the only major cloud provider without a serverless Postgres offering. As of today AWS has DSQL: aws.amazon.com/about-aws/wh... Azure partnered with Neon in August: neon.tech/blog/neon-is...

Announcing Amazon Aurora DSQL (Preview) - AWS

Discover more about what's new at AWS with Announcing Amazon Aurora DSQL (Preview)

aws.amazon.com

IT'S HERE! Amazon Aurora Distributed SQL (DSQL). It's DynamoDB and Postgres merged into a truly serverless relational database. This is THE announcement of Re:Invent 2024. Everything else is irrelevant.

Announcing Amazon Aurora DSQL (Preview)

AWS introduces Amazon Aurora DSQL, a serverless, distributed SQL database offering unlimited scalability, high availability, and zero infrastructure management...

buff.ly

S3 gets *native* Iceberg support. 🤯 From the AWS blog: aws.amazon.com/blogs/aws/ne... We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.

New Amazon S3 Tables: Storage optimized for analytics workloads | Amazon Web Services

Amazon S3 Tables optimize tabular data storage (like transactions and sensor readings) in Apache Iceberg, enabling high-performance, low-cost queries using Athena, EMR, and Spark.

aws.amazon.com

Nikhil Benesch@benesch.bsky.social · 2y ago

Something interesting is brewing in Iceberg-on-S3 land. 👀 lists.apache.org/thread/v7x65... cc @eatonphil.bsky.social

@glaubercosta.bsky.social nails an explanation of 1) why vector databases are mostly a feature not a product but 2) why nonetheless @turbopuffer.bsky.social is on to something. From a Software Huddle interview a few months back: www.youtube.com/watch?v=a4X7...

SQLite, Turso, and the State of Databases with Glauber Costa

YouTube video by Software Huddle

youtube.com

@fasterthanli.me what's the performance of loona like? 👀 I'm in the market for a faster S3 client in Rust and loona is the first plausible project I surfaced. bsky.app/profile/bene...

Nikhil Benesch@benesch.bsky.social · 2y ago

Does it use io_uring? All I want for Christmas is an S3 client that uses io_uring.

Awesome work from @antiguru.bsky.social that can cut @materialize.com memory use by 50% on a common class of workload. Most importantly for our customers: Materialize is now 50% cheaper on these workloads!

Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish. materialize.com/changelog/20...

Somehow Starlark inherited all the same Unicode problems that Python 2 had. 🙃 github.com/tidbyt/pixle... cost me more time than I would like to admit.

Add a .bytes() method to HTTP responses by benesch · Pull Request #1095 · tidbyt/pixlet

The new .bytes() method returns the body as bytes rather than a string. It is modeled after the .json() method, which parses the body as JSON. The use case is parsing HTTP requests that return bina...

github.com

Something interesting is brewing in Iceberg-on-S3 land. 👀 lists.apache.org/thread/v7x65... cc @eatonphil.bsky.social

Annoying detail about S3's new `If-Match` implementation: these writes can fail with 409 Conflict (docs: docs.aws.amazon.com/AmazonS3/lat...) I guess you're just meant to retry, but it's not clear to me why S3 wouldn't just retry internally. The whole point is that the requests are now idempotent!

S3 has finally added `put-if-match` for compare-and-swap! Expect AWS turbopuffer regions very soon aws.amazon.com/about-aws/wh...

Amazon S3 adds new functionality for conditional writes - AWS

Discover more about what's new at AWS with Amazon S3 adds new functionality for conditional writes

aws.amazon.com

Mission accomplished! I’ll be living in luxury with eight (8!) upcoming subway arrivals displayed on my Tidbyt. Developing for the Tidbyt was a mostly pleasant experience. Needing to hand roll a protobuf parser in Starlark was a curveball though: github.com/benesch/tidb...

Working on the important stuff today: teaching my Tidbyt to display subway arrival times for more than two trains at a time. Here’s what the current app looks like.