Some (really) big AI models work like a team of experts. For each piece of text, a router picks one expert to handle it. Only that path runs (so less compute needed!). That is Switch-style Mixture of Experts: more capacity without using the whole model every time.

CLIP in one picture: put images and captions in the same batch, encode both, make the matching pairs win the similarity matrix. The diagonal is supervision. Everything off the diagonal becomes a negative example!

Model Soups are one of those results that feel like they should not work, but somehow often do 🤷 Start from one base model, fine-tune it several times, then simply average the matching weights. Surprisingly, the soup model can outperform individual ones, while still being just 1 model at inference

Self-attention was popularized in language, then spread almost everywhere in machine learning: vision, audio, biology, robotics… The core idea is simple: each token asks which other tokens matter right now. It scores them, turns the scores into soft weights, and mixes information accordingly.

Fine-tuning a whole LLM is often wasteful when the pretrained weights already know most of the work. LoRA freezes the big weight matrix and learns a small low-rank update around it. Same forward pass shape, but the task-specific part becomes tiny.

Position in transformers is not always added as a learned embedding. RoPE injects position by rotating query and key vectors according to each token’s place in the sequence. This makes attention scores depend on relative distance, one reason RoPE became standard in modern LLMs.

Modern generative models often optimize through random variables (VAEs, diffusion models...). The reparameterization trick rewrites sampling as parameters plus external noise passed through a deterministic function. Same stochastic computation, but now gradients have a path through it!

That guidance scale slider in diffusion models is not magic prompt strength! Classifier-free guidance compares an unconditional denoising direction with a prompt-conditioned one, then amplifies the difference. More prompt control, until the image gets brittle.

LLMs do not sample from every possible next token. Nucleus sampling (top-p) keeps the smallest high-probability set whose mass passes p, then samples inside it. Less greedy than argmax, less chaotic than sampling from the full distribution.

🚨 New paper on regression and classification! Adding to the discussion on using least-squares or cross-entropy, regression or classification formulations of supervised problems! A thread on how to bridge these problems:

🚀 Policy gradient methods like DeepSeek’s GRPO are great for finetuning LLMs via RLHF. But what happens when we swap autoregressive generation for discrete diffusion, a rising architecture promising faster & more controllable LLMs? Introducing SEPO ! 📑 arxiv.org/pdf/2502.01384 🧵👇

Beautiful work!!

Ambroise Odonnat @ambroiseodt.bsky.social · 2y ago

🚀Proud to share our work on the training dynamics in Transformers with Wassim Bouaziz & @viviencabannes.bsky.social @Inria @MetaAI 📝Easing Optimization Paths arxiv.org/pdf/2501.02362 (accepted @ICASSP 2025 🥳) 📝Clustering Heads 🔥https://arxiv.org/pdf/2410.24050 🖥️ github.com/facebookrese... 1/🧵

🚀Proud to share our work on the training dynamics in Transformers with Wassim Bouaziz & @viviencabannes.bsky.social @Inria @MetaAI 📝Easing Optimization Paths arxiv.org/pdf/2501.02362 (accepted @ICASSP 2025 🥳) 📝Clustering Heads 🔥https://arxiv.org/pdf/2410.24050 🖥️ github.com/facebookrese... 1/🧵

Happy to see Disentangled In-Context Learning accepted at ICLR 2025 🥳 Make zero-shot reinforcement learning with LLMs go brrr 🚀 🖥️ github.com/abenechehab/... 📜 arxiv.org/pdf/2410.11711 Congrats Abdelhakim (abenechehab.github.io) for leading it, always fun working with nice and strong people 🤗

GitHub - abenechehab/dicl: Official implementation of DICL (Disentangled In-Context Learning), featured in the paper Zero-shot Model-based Reinforcement Learning using Large Language Models.

Official implementation of DICL (Disentangled In-Context Learning), featured in the paper Zero-shot Model-based Reinforcement Learning using Large Language Models. - abenechehab/dicl

github.com

For the French-speaking audience, S. Mallat's courses at the College de France on Data generation in AI by transport and denoising have just started. I highly recommend them, as I've learned a lot from the overall vision of his courses. Recordings are also available: www.youtube.com/watch?v=5zFh...

Génération de données en IA par transport et débruitage (1) - Stéphane Mallat (2024-2025)

YouTube video by Mathématiques et informatique - Collège de France

youtube.com

Speculative sampling accelerates inference in LLMs by drafting future tokens which are verified in parallel. With @vdebortoli.bsky.social , A. Galashov & @arthurgretton.bsky.social , we extend this approach to (continuous-space) diffusion models: arxiv.org/abs/2501.05370

The idea that one needs to know a lot of advanced math to start doing research in ML seems so wrong to me. Instead of reading books for weeks and forgetting most of them a year later, I think it's much better to try do things, see what knowledge gaps prevent you from doing them, and only then read.

This seems like… what we started with, no? arxiv.org/abs/2410.02724

Large Language Models as Markov Chains

Large language models (LLMs) have proven to be remarkably efficient, both across a wide range of natural language processing tasks and well beyond them. However, a comprehensive theoretical analysis o...

arxiv.org

🚨So, you want to predict your model's performance at test time?🚨 💡Our NeurIPS 2024 paper proposes 𝐌𝐚𝐍𝐨, a training-free and SOTA approach! 📑 arxiv.org/pdf/2405.18979 🖥️https://github.com/Renchunzi-Xie/MaNo 1/🧵(A surprise at the end!)

I wrote a summary of the main ingredients of the neat proof by Hugo Lavenant that diffusion models do not generally define optimal transport. github.com/mathematical...

🚀 Did you know you can use the in-context learning abilities of an LLM to estimate the transition probabilities of a Markov chains? The results are pretty exciting ! 😄