Michael Kirchhof
@mkirchhof.bsky.social
RS on uncertainty quantification for agents at Apple
🚀 We're hiring a Research Scientist, Machine Learning Research (MLR), Barcelona ☀️ 👉 Apply here: jobs.apple.com/en-us/detail... #MachineLearning #MLResearch #AI #Hiring #Apple #Barcelona #ResearchScientist #DeepLearning
AIML - Machine Learning Researcher - MLR - Jobs - Careers at Apple
Apply for a AIML - Machine Learning Researcher - MLR job at Apple. Read about the role and find out if it’s right for you.
jobs.apple.com
Chat with me about uncertainty + LLM (agents) at #ICLR2026, in either of these 6 sessions: 1. Thu 10:30 AM - 1 PM, Pavilion 3 P3-#309, Memory LLM paper 2. Thu 3:15 PM – 5:45 PM, Pavilion 3 P3-#312, SelfReflect 3. Fri, 9:30 AM - 11:30 AM, Apple booth
Michael Kirchhof@mkirchhof.bsky.social · 8mo ago4 ICLR papers 🥳 There’s an insightful story between them: If you sample LLMs multiple times, they are calibrated, even on higher levels [1], but they cannot talk about this uncertainty in a single prompt [2], so you have to help them out to gather information Bayes-optimally [3]
Advice for ICLR (or any other conference): Don't collect posters. Collect understanding. 🧵
New paper 🥳 RL relies a lot on an agent’s capability to explore. Our strategy-guided exploration makes the agent find new solutions more efficiently. It learns faster, and in some environments its Pass@1 surpasses the base model’s Pass@128. 🧵 1/6 📄 arxiv.org/abs/2603.02045
Deciding which tokens are learnable and which not is not just a question of loss. Check out our new foundational research paper on training small language models that remain factual 👇
@szilviaujvary.bsky.social · 7mo agoSmall Language Models (SLMs) don’t have the capacity to remember everything in their training data. Which tokens should they learn to predict, and when should they ask for help? We tackle this question in our new preprint. You can check it out on arxiv: arxiv.org/abs/2602.12005 🧵1/7
4 ICLR papers 🥳 There’s an insightful story between them: If you sample LLMs multiple times, they are calibrated, even on higher levels [1], but they cannot talk about this uncertainty in a single prompt [2], so you have to help them out to gather information Bayes-optimally [3]
In our new work — Complete(d)P — we try to answer 3 questions about hyperparameter (HP) scaling: ● How to transfer across model size, tokens&batch-size?→ Complete(d)P ● Do per-module HPs matter? ✔️2x speed-ups possible ● Do they transfer to larger scale? ✔️ With the right parameterisation
If you want some holiday reflections: This is not just a blogpost, but an insight into the philosophy of one of the best scientific minds (and best humans, really) I had the honor to share a bit of my life with.
Seong Joon Oh@coallaoh.bsky.social · 9mo ago🍎 Updated the Principles repository github.com/coallaoh/Pri... after a long break.
Our research team is hiring PhD interns 🍏 Spend your next summer in Paris and explore the next frontiers of LLMs for uncertainty quantification, calibration, RL and post-training, and Bayesian experimental design. Details & Application ➡️ jobs.apple.com/en-my/detail...
Internship - Machine Learning Research on Uncertainty - Jobs at Apple (MY)
Apply for a Internship - Machine Learning Research on Uncertainty job at Apple. Read about the role and find out if it’s right for you.
jobs.apple.com
📢 We’re looking for a researcher in in cogsci, neuroscience, linguistics, or related disciplines to work with us at Apple Machine Learning Research! We're hiring for a one-year interdisciplinary AIML Resident to work on understanding reasoning and decision making in LLMs. 🧵
We have been working with Michal Klein on pushing a module to train *flow matching* models using JAX. This is shipped as part of our new release of the OTT-JAX toolbox (github.com/ott-jax/ott) The tutorial to do so is here: ott-jax.readthedocs.io/tutorials/ne...
It's that time of the year! 🎁 The Apple Machine Learning Research (MLR) team in Paris is hiring a few interns, to do cool research for ±6 months 🚀🚀 & work towards publications/OSS. Check requirements and apply: ➡️ jobs.apple.com/en-us/detail... More❓→ ✉️ mlr_paris_internships@group.apple.com
LLMs are currently this one big parameter block that stores all sort of facts. In our new preprint, we add context-specific memory parameters to the model, and pretrain the model along with a big bank of memories. 📑 arxiv.org/abs/2510.02375 [1/10]🧵
Our two phenomenal interns, Alireza Mousavi-Hosseini and Stephen Zhang @syz.bsky.social have been cooking some really cool work with Michal Klein and me over the summer. Relying on optimal transport couplings (to pick noise and data pairs) should, in principle, be helpful to guide flow matching 🧵
Many treat uncertainty = a number. At Apple, we're rethinking this: LLMs should output strings that reveal all information of their internal distributions. We find that Reasoning, SFT, CoT can't do it - yet. To get there, we introduce the SelfReflect benchmark. arxiv.org/pdf/2505.20295
Natural idea. Looks like a nice paper too. arxiv.org/abs/2508.21184
BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design
We propose a general-purpose approach for improving the ability of Large Language Models (LLMs) to intelligently and adaptively gather information from a user or other external source using the framew...
arxiv.org
I'll present my view on the future of uncertainties in LLMs and vision models at @icmlconf.bsky.social, in penal discussions, posters, and workshops. Reach out if you wanna chat :) Here's everything from me and other folks at Apple: machinelearning.apple.com/updates/appl...
Can LLMs access and describe their own internal distributions? With my colleagues at Apple, I invite you to take a leap forward and make LLM uncertainty quantification what it can be. 📄 arxiv.org/abs/2505.20295 💻 github.com/apple/ml-sel... 🧵1/9
NEW PAPER ALERT: Recent studies have shown that LLMs often lack robustness to distribution shifts in their reasoning. Our paper proposes a new method, AbstRaL, to augment LLMs’ reasoning robustness, by promoting their abstract thinking with granular reinforcement learning.
I‘ll talk today about our latest research on uncertainty quantification at Apple (papers are 2 weeks old) and what I see as the future for UQ in vision and LLMs. See you at 102B, 4:30pm! PS: Lmk if you wanna chat :)
At the end of my PhD, I reflected on uncertainty quantification research, and what might change with chatbots and LLM agents. This was now accepted as position paper at @icmlconf.bsky.social. Some of those future topics are already picking up pace, so have an evening read ☕ arxiv.org/abs/2505.22655
Now that @interspeech.bsky.social registration is open, time for some shameless promo! Sign-up and join our Interspeech tutorial: Speech Technology Meets Early Language Acquisition: How Interdisciplinary Efforts Benefit Both Fields. 🗣️👶 www.interspeech2025.org/tutorials ⬇️ (1/2)
https://www.interspeech2025.org/tutorials
Your cookies are disabled, please enable them.
interspeech2025.org
Aleatoric and epistemic uncertainty are clear-cut concepts, right? ... right? 😵💫 In our new ICLR blogpost we let different schools of thought speak and contradict each other, and revisit chatbots where “the character of aleatory ‘transforms’ into epistemic” iclr-blogposts.github.io/2025/blog/re...
Today we have released the code and a demo iOS application for FastVLM - our extremely efficient and fast vision language model which runs on your device using MLX! You can check out the code and the app here: github.com/apple/ml-fas...
Paper🧵 (cross-posted at X): When does composition of diffusion models "work"? Intuitively, the reason dog+hat works and dog+horse doesn’t has something to do with independence between the concepts being composed. The tricky part is to formalize exactly what this means. 1/
Preetum Nakkiran@preetumnakkiran.bsky.social · 2y agofinally managed to sneak my dog into a paper: arxiv.org/abs/2502.04549
🚨 Apple Machine Learning Research Internship opportunity! My colleagues in Apple MLR are looking for a PhD research intern with a strong interest in reinforcement learning/post-training for LLMs. If interested, apply by sending an email to Etai Littwin (elittwin at apple dot com)
Wow, OpenAI's o1 has a whopping 93% ECE on Humanity's Last Exam. So if you just prompt o1 to tell you how sure it is about its answer, it will basically produce gibberish. And that's how most users will ask for uncertainties. We have work to do!
Today is a great day for optimal transport 🎉! Lots of gratitude 🙏 for all folks who contributed to ott-jax.readthedocs.io and pushed for the MOSCOT (now @ nature!) paper, from visionaries @dominik1klein.bsky.social, G. Palla, Z. Piran to the magician, Michal Klein! ❤️ www.nature.com/articles/s41...
Mapping cells through time and space with moscot - Nature
Moscot is an optimal transport approach that overcomes current limitations of similar methods to enable multimodal, scalable and consistent single-cell analyses of datasets across spatial and temporal...
nature.com
Helmholtz Munich@helmholtzmunich.bsky.social · 2y agoAI in Cell Research: Unveiling Organ Development #Moscot, a groundbreaking #AI tool developed by an international team led by #HelmholtzMunich, tracks cells in real time, revealing how organs like the #pancreas form. t1p.de/viil5 @fabiantheis.bsky.social @nature.com
Many LLM uncertainty estimators perform similarly, but does that mean they do the same? No! We find that they use different cues, and combining them gives even better performance. 🧵1/5 📄 openreview.net/forum?id=QKR... NeurIPS: Sunday, East Exhibition Hall A, Safe Gen AI workshop
Interested in learning how to evaluate uncertainty in LLMs? Check out our work at NeurIPS! Feel free to reach out for a chat!
Michael Kirchhof@mkirchhof.bsky.social · 2y agoEvaluating your LLM uncertainties with Rougle-L will show clear winners... except that they aren't actually good. We find that Rouge-L spuriously favors some methods over others. 🧵1/4 📄 openreview.net/forum?id=jGt... NeurIPS: Sunday, East Exhibition Hall A, Safe Gen AI workshop
