Shiv Gupta
@sgupta.xyz
Data Engineer @ Chainalysis. Views my own đź“ŤToronto sgupta.xyz
Shivanshu Gupta is a Data Engineer at Chainalysis, where they run Dagster to orchestrate and maintain complex data pipelines. 🔍 In today's data landscape, engineers like Shivanshu need tools that reduce cognitive load while handling increasingly complex pipelines.
S3 (Iceberg) Tables is everything I dreamt of, and more. I blogged some long-form thoughts: meltware.com/2024/12/04/s... I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets. #databs
A First Look at S3 (Iceberg) Tables
AWS announced S3 Tables today, which brings native support for Apache Iceberg to S3. It’s hard to overstate how exciting this is for the data analytics ecosystem. This post is a quick rundown of my th...
meltware.com
Anyone here try to self-host Rill (@rilldata.com) or is that not a thing? $250/mo to deploy on Rill Cloud feels prohibitive for most hobby projects... #databs
I've been seeing posts from people coming to #dataBS and getting overwhelmed and worrying they can't contribute. I get it. I still feel that sometimes. But I'm trying to use my impostor syndrome as a strength. It's permission to accept that we'll never know everything. So join in the discussion!
Anyone here manage to attach a Databricks-hosted Unity Catalog to DuckDB? github.com/duckdb/uc_ca...
GitHub - duckdb/uc_catalog: Proof-of-concept extension combining the delta extension with Unity Catalog
Proof-of-concept extension combining the delta extension with Unity Catalog - duckdb/uc_catalog
github.com
I open my Bluesky feed and it is full of really cool people saying really smart things and being really passionate about really interesting stuff. I like this version of social media, and the world. Thanks, folks.
I underestimated how far you can get with prompt engineering. Every time I thought I needed to bite the fine-tuning bullet, what I really needed was better prompts
Going from CSV > #DuckDB > Delta Lake in S3 is a breeze. Only thing that's missing is write support to Delta Lake tables natively in SQL (vs having to go through Python). And DuckDB Google Sheets integration makes me want to use it for everything
