DataLoader Dispatching When constrained by a variety of reasons to where you can't include multiple copies (or mmaps) of datasets in memory, be it too many concurrent streams, low resource availability, or a slow CPU, dispatching is here to help.

Batch Sampler Sharding

Dataset Sharding When performing distributed data parallelism, we split the dataset every batch so every device sees a different chunk of the data. There are different methods for doing so. One example is sharding at the *dataset* level, shown here.

Hi all 👋 Back very briefly to mention I’m working on a new course, and there’s a star-studded set of guest speakers 🎉 From Scratch to Scale: Distributed Training (from the ground up). From now until I’m done writing the course material, it’s 25% off :) maven.com/walk-with-co...

AAAHHHHHHHHH BE NICE TO OPEN SOURCE MAINTAINERS OH MY GOD. SOME OF YOU ARE SO RUDE, WHO RAISED YOU

JK STRONG is still awesome. Solution (which they were already aware of this) was literally logout -> login! 11/10 customer support

Zach Mueller@muellerzr.bsky.social · last yr.

Enshittification has come to my gym app. Lord help us all 😭

Enshittification has come to my gym app. Lord help us all 😭

Recently did my first lifting competition, overall quite happy where things ended up (bar bench) and was a ton of fun. Might wait a year and give it another go, but I did get first in the weight class! 145kg squat 180kg deadlift 90kg bench

I guess work is done for the day, everyone can go home 🤷‍♂️

A federal judge in Maryland issued a temporary restraining order on Monday, blocking members of Elon Musk’s so-called Department of Government Efficiency team and anyone “working on the DOGE agenda at the Department of Education” from accessing sensitive data until March 10.

Day one of new program: ✅

Nothing humbles you more than a front squat. Going from 260lbs back squat to 115lbs front squat was *lovely*

Interesting discussions between John Ousterhout and "Uncle Bob" on software design: method length, comments, and test-driven development. It's a great example of "agree to disagree". There is no silver bullet in software design. Let's focus on writing easy-to-parse code 👀 github.com/johnousterho...

Estimated read time: 2-3 days Who’s going to be the first to start a study group?

Thomas Wolf@thomwolf.bsky.social · last yr.

After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook": hf.co/spaces/nanot... A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!

After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook": hf.co/spaces/nanot... A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!

The Ultra-Scale Playbook - a Hugging Face Space by nanotron

The ultimate guide to training LLM on large GPU Clusters

hf.co

Anyone in my bubble built a lack rack before? 👀 Eyeballing doing this for the new place

0/10 can’t recommend: * Selling your house * Buying a house (*after* putting your old house on the market) * Lifting competition All in the same week

Update: Ran the equivalent of this space locally via fastapi + hooked into the LLM commit message generator got the job done, and very fast (since it’s a T5 model under the hood) huggingface.co/spaces/mamik...

Commit Message Generator - a Hugging Face Space by mamiksik

Discover amazing ML apps made by the community

huggingface.co

Zach Mueller@muellerzr.bsky.social · last yr.

Me post seeing SmolLM2 quant is fast enough to run on my tiny laptop

Me post seeing SmolLM2 quant is fast enough to run on my tiny laptop

Happy first intern day at @hf.co to all who celebrate 🤗 This years round of interns are 🔥

We are reproducing the full DeepSeek R1 data and training pipeline so everybody can use their recipe. Instead of doing it in secret we can do it together in the open! Follow along: github.com/huggingface/...

GitHub - huggingface/open-r1: Fully open reproduction of DeepSeek-R1

Fully open reproduction of DeepSeek-R1. Contribute to huggingface/open-r1 development by creating an account on GitHub.

github.com

Wrote up my selfish personal argument for releasing code as Open Source: if you solve a problem and then release it under an Open Source license you will never have to solve that problem again for the rest of your career! simonwillison.net/2025/Jan/24/...

Me: huh where’s the cell signal go. Where’d the internet go Verizon: I suppose I’m forcibly done with work for the day, too bad I had to go to the next town over to say that 😅

Anyone in my mutuals looking to hire a really good Elixir dev for their team? My father is recently having some hard times and looking for work. Very knowledgeable both there and on ML

Hi all 👋 I work in OSS, typically in the ML framework scope (transformers Trainer, accelerate, etc). Big nerd on framework design. Also into 🏋️ & anime (so those may show up in the feed occasionally). muellerzr.github.io

Me: huh, why is the chainsaw man intro playing on repeat at the gym? Apparently my year of Spotify premium expired today. Quick trip to the car and we’re back in business for deadlifts. If you pay monthly for Spotify, don’t. $12/mo == $144. Or pay $99 for an egiftcard www.amazon.com/dp/B09MWM6TS1

Recently been digging into torchao’s float8 training paradigm. It interests me more than transformerengine or MS-AMP simply because it promises near exact training is possible between it and using raw bf16 github.com/pytorch/ao/t...

ao/torchao/float8 at main · pytorch/ao

PyTorch native quantization and sparsity for training and inference - pytorch/ao

github.com