RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video Ulrich Prestel, @stefanabaumann.bsky.social, @rmsnorm.bsky.social, Björn Ommer tl;dr:nuisance variable, architectural unification, and autoregressive pose learning->explicit dynamic state handling arxiv.org/abs/2605.31535

Diffusion models treat every part of an image equally. → Same number of steps. Same compute. But images aren’t uniform. 🤔 Some regions are easy, others are hard. So why force the model to treat them the same? 🧵

Video diffusion models learn motion indirectly through pixels. But motion itself is much lower-dimensional. We introduce 64× temporally compressed motion embeddings that directly capture scene dynamics. This enables efficient planning -> 10,000× faster than video models. 🧵👇

Great to see corroborating evidence to our motion-forecasting.github.io work from other groups! Check out this concurrent great work from @stefanabaumann.bsky.social, @jannik-w.bsky.social, @tommymarto.bsky.social, Mahdi M. Kalayeh, and Björn Ommer (@compvis.bsky.social)

Stefan Baumann@stefanabaumann.bsky.social · 4mo ago

You don't imagine the future by mentally rendering a movie. You trace how things move -- abstractly, sparsely, step by step. We built a model that does exactly this. It predicts motion, not pixels -- and it's 3,000× faster than video world models. Myriad, accepted at @cvprconference.bsky.social

You don't imagine the future by mentally rendering a movie. You trace how things move -- abstractly, sparsely, step by step. We built a model that does exactly this. It predicts motion, not pixels -- and it's 3,000× faster than video world models. Myriad, accepted at @cvprconference.bsky.social

Last year Molmo set SOTA on image benchmarks + pioneered image pointing. Millions of downloads later, Molmo 2 brings Molmo’s grounded multimodal capabilities to video 🎥—and leads many open models on challenging industry video benchmarks. 🧵

Oof

RoMa v2 is now out! (github.com/Parskatt/rom..., arxiv.org/abs/2511.15706) Here are the main improvements we made since RoMa:

Excited to share that we'll be presenting four papers at the main conference at ICCV 2025 this week! Come say hi in Honolulu! 👋 Pingchuan, Ming, Felix, Stefan, Timy, and Björn Ommer will be attending.

🤔 What if you could generate an entire image using just one continuous token? 💡 It works if we leverage a self-supervised representation! Meet RepTok🦎: A generative model that encodes an image into a single continuous latent while keeping realism and semantics. 🧵 👇

🤔 What happens when you poke a scene — and your model has to predict how the world moves in response? We built the Flow Poke Transformer (FPT) to model multi-modal scene dynamics from sparse interactions. It learns to predict the 𝘥𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘪𝘰𝘯 of motion itself 🧵👇

“Everyone knows” what an autoencoder is… but there's an important complementary picture missing from most introductory material. In short: we emphasize how autoencoders are implemented—but not always what they represent (and some of the implications of that representation).🧵

I'm calling it now, GSPO will be the next big hype in LLM RL algos after GRPO. It makes so much more sense intuitively to work on a sequence rather than on a token level when our rewards are on a sequence level.

🎉 Excited to share that our lab has three papers accepted at CVPR 2025! Come say hi in Nashville! 👋 Johannes, Ming, Kolja, Stefan, and Björn will be attending.

If you are interested, feel free to check the paper (arxiv.org/abs/2506.02221) or come by at CVPR: 📌 Poster Session 6, Sunday 4:00 to 6:00 PM, Poster #208

Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment

Diffusion models have revolutionized generative tasks through high-fidelity outputs, yet flow matching (FM) offers faster inference and empirical performance gains. However, current foundation FM mode...

arxiv.org

Here's the third and final part of Slater Stich's "History of diffusion" interview series! The other two interviewees' research played a pivotal role in the rise of diffusion models, whereas I just like to yap about them 😬 this was a wonderful opportunity to do exactly that!

History of Diffusion - Sander Dieleman

YouTube video by Bain Capital Ventures

youtube.com

#KostasThoughts: Another major conference review drop is around the corner. In baseball, a .300 average is elite. In research, it’s a familiar reality: submitting to top conferences means rejections happen. Keep swinging!

I am very happy to share our latest work on the information theory of generative diffusion: "Entropic Time Schedulers for Generative Diffusion Models" We find that the conditional entropy offers a natural data-dependent notion of time during generation Link: arxiv.org/abs/2504.13612

New blog post: let's talk about latents! sander.ai/2025/04/15/l...

Generative modelling in latent space

Latent representations for generative models.

sander.ai

And the CVPR oral decisions are out! (on Openreview)

Introducing VGGT (CVPR'25), a feedforward Transformer that directly infers all key 3D attributes from one, a few, or hundreds of images, in seconds! Project Page: vgg-t.github.io Code & Weights: github.com/facebookrese...

Introducing DaD (arxiv.org/abs/2503.07347), a pretty cool keypoint detector. As this will get pretty long, this will be two threads. The first will go into the RL part, and the second on the emergence and distillation.

The fate of your #CVPR2025 submission

🤔When combining Vision-language models (VLMs) with Large language models (LLMs), do VLMs benefit from additional genuine semantics or artificial augmentations of the text for downstream tasks? 🤨Interested? Check out our latest work at #AAAI25: 💻Code and 📝Paper at: github.com/CompVis/DisCLIP 🧵👇

What I like to do when considering a new dataset is to train a simple classifier and look at 'the most confident errors'. Recently with NICO: Apart from a class, the images have a context, one of them is 'autumn'. There is also a pumpkin class. Surprise surprise, many autumn images contain pumpkins.

Just had an idea

How to schedule a meeting? When you ask for a meeting with others, you are asking for their time. You are asking for their most valuable, finite resource to benefit yourself (e.g., for advice, networking, questions, and opportunities). Here are some tips that I found useful.

Do you like the power of diffusion features for semantic correspondence but dread running an expensive ~1B model to get them? What if you could have even better features at a fraction of the cost? If this sounds enticing, take a look at this paper! ⬇️

Frank Fundel@frankfundel.bsky.social · 2y ago

Did you know you can distill the capabilities of a large diffusion model into a small ViT? ⚗️ We showed exactly that for a fundamental task: semantic correspondence📍 A thread 🧵👇

Ever wondered if diffusion features could do better without all the noise? 🤔 Turns out they can! We show how adapting the backbone unlocks clean, powerful features for better results across the board. 🚀🧹 Check it out! ⬇️

Nick Stracke@rmsnorm.bsky.social · 2y ago

🤔 Why do we extract diffusion features from noisy images? Isn’t that destroying information? Yes, it is - but we found a way to do better. 🚀 Here’s how we unlock better features, no noise, no hassle. 📝 Project Page: compvis.github.io/cleandift 💻 Code: github.com/CompVis/clea... 🧵👇

Blog post link: diffusionflow.github.io/ Despite seeming similar, there is some confusion in the community about the exact connection between the two frameworks. We aim to clear up the confusion by showing how to convert one framework to another, for both training and sampling.