Tessera
Folgen

Sean Lopp

@loppsean.bsky.social

Dagster, RStudio, Looker. Dad. Bikes. Coffee.

70 Posts597 Follower322 Folgt
Christian Minich@christianminich.com · 2y ago

@loppsean.bsky.social's walk through of asset checks in Day 11 is awesome, do check it out! In some cases, you might not want to re-read in the asset data (or you might even want to compare across previous materializations!) In my example, I use Dagster's metadata system to do just that.

@dg.asset( deps=[csv_external_asset], automation_condition=dg.AutomationCondition.on_cron("* * * * *"), ) def orders(context: dg.AssetExecutionContext, csv_storage: CsvStorageResource) -> None: orders = csv_storage.read_data('orders_raw.csv') context.log.info(f"Creating asset one with data: {orders.head()}") # directly write csv to storage csv_storage.write_data(orders, 'orders.csv') return dg.MaterializeResult( metadata={ "dagster/row_count": dg.MetadataValue.int(len(orders)), "preview": dg.MetadataValue.md(orders.head().to_markdown()), "dagster/column_schema": create_table_schema_metadata_from_dataframe(orders) } ) @dg.asset_check(asset="orders") def orders_increasing_or_equal(context: dg.AssetCheckExecutionContext) -> dg.AssetCheckResult: histotical_metadata = context.instance.fetch_materializations( dg.AssetRecordsFilter( asset_key=dg.AssetKey("orders"), ), limit=2 ) if len(histotical_metadata) < 2: return dg.AssetCheckResult(passed=True) row_count_values = [ record.asset_event.metadata.get("dagster/row_count") for record in histotical_metadata[0] ] latest_row_count = row_count_values[0] previous_row_count = row_count_values[1] return dg.AssetCheckResult( passed=latest_row_count>=previous_row_count, metadata={"latest_row_count": latest_row_count, "previous_row_count": previous_row_count} )dagster asset check run showing latest row count of 99 and current row count of 100
Sean Lopp@loppsean.bsky.social · 2y ago

Day 11! The diff between orchestration and data orchestration? Your code can stay the same, still work, but the data can change and break everything! youtu.be/ugS1KQhxWrA

Sean Lopp@loppsean.bsky.social · 2y ago

Me trying to figure out Power BI pricing To share content and collaborate in Microsoft Fabric, your organization needs to have an F or P capacity, and at least one per-user license.

Sean Lopp@loppsean.bsky.social · 2y ago

I've spent time in all 3 cloud provider consoles, unequivocally Azure is the worst. I don't mean to offend ... If I am.... DM me. Maybe you can show me the ropes? Example: the organization > account > subscription hierarchy, when applied to roles, is insane. I am sudo. Respect my authority!

Sean Lopp@loppsean.bsky.social · 2y ago

There are two types of Git commit messages: 8:00am: ae678pt: "as described in the linked RFC, implementing changes to the partition strategy to reduce hotspots in our cluster index" 4:49pm: stcm98: "f-ing typos, try again. pleeeeease pass"

Sean Lopp@loppsean.bsky.social · 2y ago

Another DOGE suggestion Require 401k providers to give you the option to initiate a rollover to another firm without talking to their shitty sales team and _faxing_ forms.