Incremental Jobs and Data Quality Are On a Collision Course - Part 1 - The Problem — Jack Vanlightly

Big data isn’t dead; it’s just going incremental If you keep an eye on the data space ecosystem like I do, then you’ll be aware of the rise of DuckDB and its message that big data is dead. The idea comes from two industry papers (and associated data sets), one from the Redshift team (paper and dataset) and one from Snowflake (paper and dataset). Each paper analyzed the queries run on their platforms, and some surprising conclusions were drawn – one being that most queries were run over quite small data. The conclusion (of DuckDB) was that big data was dead, and you could use simpler query engines rather than a data warehouse. It’s far more nuanced than that, but data shows that most queries are run over smaller datasets.  Why?

jack-vanlightly.com

davidasboth.com

"Tidy First?" by Kent Beck

Kent Beck’s “Tidy First?" is a concise and engaging read, outlining several “tidyings”–small code improvements–that make software easier to understand and more adaptable to future changes. He emphasiz...

andrewfisher.me

Feldera: Bridging Batch and Streaming with Incremental Computation

Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous…

dataengineeringpodcast.com