Robert Nishihara

@robertnishihara.bsky.social

Co-founder of Anyscale. Co-creator of Ray. Previously PhD ML at Berkeley.

DeepSeek released smallpond, a big data processing framework built on top of Ray. - Smallpond targets high performance data processing. - It provides a high-level dataframe API - Targets petabyte-level scaling The challenges around training data prep only grow when you include multimodal data.

Bild

Amazon published this only 4 months ago, but it feels like an eternity. It's one of the most impressive large-scale data processing migration efforts. Rare to see companies truly achieving order of magnitude cost improvements (while simultaneously increasing scale). aws.amazon.com/blogs/openso...

Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 | Amazon Web Services

Large-scale, distributed compute framework migrations are not for the faint of heart. There are backwards-compatibility constraints to maintain, performance expectations to meet, scalability limits to...

aws.amazon.com