Tessera
Folgen

Kit Menke

@kitmenke.com

Data Engineering leader in Saint Louis, STL Big Data I.D.E.A. meetup organizer, lifelong learner and teacher. He / him #dataBS

59 Posts377 Follower396 Folgt
Crystal Lewis@cghlewis.bsky.social · 10mo ago

I just want everyone to know that if we are connected on here, I’m rooting for you. For your career, your creative endeavor, your happiness, whatever you are chasing. I hope it happens for you!

rmoff 🏃‍♂️🫖🥓@rmoff.net · 2y ago

Couple of big announcements from @cloudflare.social today for folk in #dataBS: * Acquisition of Arroyo, launch of Pipelines for streaming ingestion: blog.cloudflare.com/cloudflare-a... * Launch of R2 Data Catalog—a managed Apache Iceberg catalog for R2 blog.cloudflare.com/r2-data-cata...

Just landed: streaming ingestion on Cloudflare with Arroyo and Pipelines

We’ve just shipped our new streaming ingestion service, Pipelines — and we’ve acquired Arroyo, enabling us to bring new SQL-based, stateful transformations to Pipelines and R2.

blog.cloudflare.com

Matthew Mullins@mmullins.coginiti.co · 2y ago

I found this while looking through some scratch notes. I don't now recall what the context was, but it's an interesting thought on the evolution of the data warehouse. (Though there is an equivocation imbedded in this history) #databs

1988 Data Warehouse Architecture is introduced 1994 Data Warehouse is too difficult to build -> Data Marts 2008 Data Warehouse is too small for big data -> Hadoop 2010 Data Warehouse is too structured -> Data Lake 2012 Data Warehouse is too difficult to scale -> Cloud Data Warehouse 2016 Data Warehouse is too difficult to manage -> Data Fabric 2019 Data Warehouse is too centralized -> Data Mesh 2020 Data Warehouse is too limiting -> Data Lakehouse 2023 Data Warehouse is too monolithic -> Composable Data Platform
Kelsey Hightower@kelseyhightower.com · 2y ago

1. Impact. How much revenue does my work protect or generate? 2. Quality. Does my work meet or exceed customer expectations? 3. Efficiency. Reward making the right buy versus build decision. 4. Reusability. How do others leverage my work? 5. Supportability. How much work do I create for others?

Erica Joy Astrella@ericajoy.astrel.la · 2y ago

a question for the people who write code for money: if you could wave a magic wand and have your performance/promotability measured on any 5 metrics of your choice, what would those metrics be?

David Jayatillake@jayatillake.bsky.social · 2y ago

The more I think about yesterday's announcement about Amazon S3 Tables, the more I think that it changes things a great deal. The gravity of data has shifted from the warehouse to cloud storage... but is there really a difference any more? 🧵1/n www.businesswire.com/news/home/20....

Amazon S3 Expands Capabilities with Managed Apache Iceberg Tables for Faster Data Lake Analytics and Automatic Metadata Generation to Simplify Data Discovery and Understanding

At AWS re:Invent, Amazon Web Services, Inc. (AWS), an Amazon.com, Inc. company (NASDAQ: AMZN), today announced new Amazon Simple Storage Service (Amaz

businesswire.com

Kit Menke@kitmenke.com · 2y ago

In two weeks, the St. Louis Big Data I.D.E.A. meetup is hosting @chad-isenberg.bsky.social to give an overview on dbt, alternatives, and the future of "the last mile" in data management. Beginners welcome! 🗓️ When: December 4, 2024 @ 5:30 PM 📍Where: Virtually on Zoom RSVP below! #dataBS #datasky

[VIRTUAL] A Brief Introduction to dbt, Wed, Dec 4, 2024, 5:30 PM | Meetup

In this talk, we'll cover what dbt is, why it's useful, alternatives, and the future of "the last mile" in data management. I'll assume folks have no knowledge of dbt and m

meetup.com

Simon Willison@simonwillison.net · 2y ago

Foursquare just open sourced their 100 million place point of interest dataset! Some notes on poking around with it using DuckDB (it's Parquet files on S3) simonwillison.net/2024/Nov/20/...

Foursquare Open Source Places: A new foundational dataset for the geospatial community

I did not expect this! > [...] we are announcing today the general availability of a foundational open data set, Foursquare Open Source Places ("FSQ OS Places"). This base layer …

simonwillison.net

Theodore Manassis@mamonu.bsky.social · 2y ago

Some work that I have been involved in the last year. I hope you like the blogpost from our lead, Soumaya as its a very interesting solution. Not all problems are nails to the hammer of Spark :) ministryofjustice.github.io/data-and-ana...

Building a transaction data lake using Amazon Athena, Apache Iceberg and dbt

How we leveraged Amazon Athena, along with the Apache Iceberg table format and the dbt SQL management framework, to build robust, scalable and maintainable ELT (extract, load, transform) pipelines.

ministryofjustice.github.io

Martin Kleppmann@martin.kleppmann.com · 2y ago

In June Elon posted this graph of the rate of likes on X. It doesn't have a unit on the y axis, but it's plausible to assume that it's events/sec. If that is true, X in June was handling about 20k likes/sec. For comparison, Bluesky is now handling about 700 likes/sec during the busy part of the day.

A graph showing the rate of likes on X over the course of a couple of hours on 12 June 2024. It fluctuates between about 17k and 20k; the y axis doesn't have a unit. A second line shows a comparison to the prior week, which is slightly lower. There are strange unexplained spikes on the hour every hour.A graph showing the rate of various types of events on Bluesky over the last day. At peak there are about 800 follow events and 700 like events per second (note the likes are scaled down by a factor of 5; the right-hand y axis is for likes).
Kit Menke@kitmenke.com · 2y ago

Thinking about creating a STL Data starter pack for those doing cool stuff in data engineering, data analytics, and data science...