Birchlabs
@birchlabs.co.uk
ML Engineer at Anlatan (NovelAI). co-author of HDiT (Hourglass Diffusion Transformers). works on diffusion models and LLMs. 日本語を勉強してる。
pytorch 2.7 is out! - Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too - foreach map looks good for optimizers - trainable biases means you can now train T5 on flex - prologue fusion is hype - context parallel brings ring attention pytorch.org/blog/pytorch...
pytorch 2.6 is out! highlights: - flex attention: better compilation of blockmask creation, better support for dynamic shapes - cuDNN SDPA: fixes for memory layout - CUDA 12.6 - python 3.13 - MaskedTensor memory leak fix
pytorch 2.6 final RC is out, promoting to stable in a couple of days! mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes. there's also fixes for memory layout in cuDNN SDPA. dev-discuss.pytorch.org/t/pytorch-re...
drink cups should put the hole in the bottom. heat rises. "the top is cool enough to drink" implies "everything below it is colder". drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.
running npm version from a subdirectory of a git repository is literally an unsolved problem in 2024 github.com/npm/cli/issu...
I should just get this tattooed, I never remember how to find it pip install huggingface_hub[hf_transfer] HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download
when the standard library comments out std::experimental::observer_ptr just to stop you having fun
when you're measuring torch compile warmup "oh 11 secs that's not so bad" then you realize it was 111 secs
if you care about multiprocess debugging in VSCode please upvote this issue so we don't have to click terminate a hundred times github.com/microsoft/vs...
Meta releases flow-matching code github.com/facebookrese...
GitHub - facebookresearch/flow_matching: A PyTorch library for implementing flow matching algorithms, featuring continuous and discrete flow matching implementations. It includes practical examples fo...
A PyTorch library for implementing flow matching algorithms, featuring continuous and discrete flow matching implementations. It includes practical examples for both text and image modalities. - fa...
github.com
the EDM2 repository had an Autoguidance update 4 days ago demonstrating their NeurIPS Oral paper "Guiding a Diffusion Model with a Bad Version of Itself" github.com/NVlabs/edm2
GitHub - NVlabs/edm2: EDM2 and Autoguidance -- Official PyTorch implementation
EDM2 and Autoguidance -- Official PyTorch implementation - NVlabs/edm2
github.com
torch profiler record_function spans are not free, even when you're not profiling a model. the model I'm benchmarking trained 2.5% faster when I commented them all out.
🤔 Why do we extract diffusion features from noisy images? Isn’t that destroying information? Yes, it is - but we found a way to do better. 🚀 Here’s how we unlock better features, no noise, no hassle. 📝 Project Page: compvis.github.io/cleandift 💻 Code: github.com/CompVis/clea... 🧵👇
torch.compile is hard for dynamic shapes / large number of static shapes, and non-transformer architectures. I measure suites of shapes, log recompiles, check which require warmup. operation compile competitive with whole-model compile. some operations prefer compiler disabled. dynamic often slow.
A common question nowadays: Which is better, diffusion or flow matching? 🤔 Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.
mood: adding unused variables to make torch inductor compile my triton kernel
that feel when you’ve been cooking rice for 10 minutes but the hob wasn’t turned on
if you get KO'd in smash, do you die? what's the safest stage to be KO'd on? Great Bay looks alright if you're a confident swimmer…
NATTEN just added fused support for self-cross attention! so you can attend to local neighbourhood and registers or text condition. it lets you reduce partial attention results (e.g. logsumexp provided by xformers APIs) into its LSE. github.com/SHI-Labs/NAT...
Support for fused cross-NA by alihassanijr · Pull Request #182 · SHI-Labs/NATTEN
Adds experimental support for additional context tokens to Fused NA. Any number of partial attention results can be reduced into a final one as if their contexts were merged, which is just the same...
github.com
