How do you know if your training recipe is ready to be scaled up? Measure effective and propagating updates. Read about these tools in our latest blog post! sbordt.substack.com/p/how-to-mea...
Sebastian Bordt
@sbordt.bsky.social
Language models and interpretable machine learning. Postdoc @ Uni Tübingen. https://sbordt.github.io/
Over at 3quarksdaily, there are two thought provoking pieces by Dwight Furrow about the AI-consciousness debate: 3quarksdaily.com/3quarksdaily... 3quarksdaily.com/3quarksdaily...
3 Quarks Daily - Science Arts Philosophy Politics Literature
Science Arts Philosophy Politics Literature
3quarksdaily.com
ah, a new possible addition to the canon of SIGBOVIK AI papers
Our spotlight paper is happening today at the #NeurIPS poster session! Drop by if you want to chat about the nitty-gritty details of large-scale transformer training!
Leena C Vankadara@leenacvankadara.bsky.social · 8mo ago📄 Paper: arxiv.org/abs/2505.22491 Catch our Spotlight at #NeurIPS2025 Today! 📅 Wed Dec 3 🕟 4:30 - 7:30 PM 📍 Exhibit Hall C,D,E — Poster #3903 Huge thanks to my amazing collaborators: @mohaas.bsky.social @sbordt.bsky.social @ulrikeluxburg.bsky.social
📄 Paper: arxiv.org/abs/2505.22491 Catch our Spotlight at #NeurIPS2025 Today! 📅 Wed Dec 3 🕟 4:30 - 7:30 PM 📍 Exhibit Hall C,D,E — Poster #3903 Huge thanks to my amazing collaborators: @mohaas.bsky.social @sbordt.bsky.social @ulrikeluxburg.bsky.social
On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling
Scaling limits, such as infinite-width limits, serve as promising theoretical tools to study large-scale models. However, it is widely believed that existing infinite-width theory does not faithfully ...
arxiv.org
Ever wondered about the rationale behind transformer training details like qk-norm, learning rate, and z-loss? Read this blog post to find out more!
Why Can We Train Large Models with Large Learning Rates?
Infinite-width theory may explain the training dynamics of finite-width neural networks after all.
open.substack.com
Here is a formal impossibility result for XAI: Informative Post-Hoc Explanations Only Exist for Simple Functions. I'll give an online presentation about this work next tuesday in @timvanerven.nl 's Theory of Interpretable AI Seminar: arxiv.org/abs/2508.11441 tverven.github.io/tiai-seminar/
🚨 Workshop on the Theory of Explainable Machine Learning Call for ≤2 page extended abstract submissions by October 15 now open! 📍 Ellis UnConference in Copenhagen 📅 Dec. 2 🔗 More info: sites.google.com/view/theory-... @gunnark.bsky.social @ulrikeluxburg.bsky.social @emmanuelesposito.bsky.social
Theory of XAI Workshop, Dec 2, 2025
Explainable AI (XAI) is now deployed across a wide range of settings, including high-stakes domains in which misleading explanations can cause real harm. For example, explanations are required by law ...
sites.google.com
I am hiring PhD students and/or Postdocs, to work on the theory of explainable machine learning. Please apply through Ellis or IMPRS, deadlines end october/mid november. In particular: Women, where are you? Our community needs you!!! imprs.is.mpg.de/application ellis.eu/news/ellis-p...
We need new rules for publishing AI-generated research. The teams developing automated AI scientists have customarily submitted their papers to standard refereed venues (journals and conferences) and to arXiv. Often, acceptance has been treated as the dependent variable. 1/
This new center strikes the right tone in approaching the AI alignment problem. alignmentalignment.ai
Center for the Alignment of AI Alignment Centers
We align the aligners
alignmentalignment.ai
A new recording of our FridayTalks@Tübingen series is online! How much can we forget about Data Contamination? by @sbordt.bsky.social Watch here: youtu.be/T9Y5-rngOLg
How much can we forget about Data Contamination? - [Sebastian Bordt]
YouTube video by Friday Talks Tübingen
youtu.be
I'm at #ICML in Vancouver this week, hit me up if you want to chat about pre-training experiments or explainable machine learning. You can find me at these posters: Tuesday: How Much Can We Forget about Data Contamination? icml.cc/virtual/2025...
Our #ICML position paper: #XAI is similar to applied statistics: it uses summary statistics in an attempt to answer real world questions. But authors need to state how concretely (!) their XAI statistics contributes to answer which concrete (!) question! arxiv.org/abs/2402.02870
Sebastian Bordt@sbordt.bsky.social · last yr.During the last couple of years, we have read a lot of papers on explainability and often felt that something was fundamentally missing🤔 This led us to write a position paper (accepted at #ICML2025) that attempts to identify the problem and to propose a solution. arxiv.org/abs/2402.02870 👇🧵
During the last couple of years, we have read a lot of papers on explainability and often felt that something was fundamentally missing🤔 This led us to write a position paper (accepted at #ICML2025) that attempts to identify the problem and to propose a solution. arxiv.org/abs/2402.02870 👇🧵
Have you ever wondered whether a few times of data contamination really lead to benchmark overfitting?🤔 Then our latest #ICML paper about the effect of data contamination on LLM evals might be for you!🚀 Paper: arxiv.org/abs/2410.03249 👇🧵
In explainable machine learning, we mostly have negative results for what post-hoc explanations cannot do. This work presents a surprisingly strong positive result for SHAP, showing that a simple sampling modification allows to reliably detect features that don't influence the model.
Ulrike Luxburg@ulrikeluxburg.bsky.social · last yr.Ever aggregated SHAP values across sample points? Our #COLT2025 paper proves that this might be safe when your goal is to discard unimportant features - but only if you add one extra line of code that reshuffles your data! With Robi Bhattacharjee and Karolin Frohnapfel arxiv.org/abs/2503.23111
Is the distinction between "aleatoric" and "epistemic" uncertainty practically meaningful (or even well defined) in any real sense? Aleotoric uncertainty refers to irreducible unpredictability (e.g. unrealized randomness in nature) whereas epistemic refers to model uncertainty.
I really like coding with LLMs. This week Claude & ChatGPT convinced me my code was too slow. After 2 days of investigation, I think my code is just fine. Never again will I blindly trust you with my profiler logs!🤖
I really like the new HTML preview on arxiv, but it somehow handles latex errors differently from PDF. I've been seeing lots of ICML error messages lately.
I just asked aistudio.google.com to write a review for a paper that we will submit to ICML. It's impressive. I believe with this tool, I could produce a mediocre paper review for almost any paper in less than 10 minutes (judged by the standard of reviews that we have at ML conferences).
Uhm so OpenAI actually has access to FrontierMath? epoch.ai/blog/openai-...
Clarifying the Creation and Use of the FrontierMath Benchmark
We clarify that OpenAI commissioned Epoch AI to produce 300 math questions for the FrontierMath benchmark. They own these and have access to the statements and solutions, except for a 50-question hold...
epoch.ai
The chain of thought in DeepSeek-R1 is pretty impressive.
ICML 2025 has some exciting changes. Here are two of my favorites. 1. Only 1 round of back-and-forth between authors & reviewers. The review process should not be an endless back and forth. It shouldn't be possible to get your paper accepted by exhausting reviewers.
Are you interested in data contamination and LLM benchmarks?🤖 Check out our poster today at the NeurIPS ATTRIB workshop (3-4:30pm)! 💡 TL;DR: In the large-data regime, a few times of data contamination matter less than you might think.
Recent works have proposed to use publicly available Kaggle competitions to benchmark LLMs, most famously OpenAI's openai.com/index/mle-be.... In this blog, I show how to test LLMs for contamination with Kaggle competitions (of course, there is contamination). sbordt.substack.com/p/data-conta...
Data Contamination in MLE-bench
How to test language models for prior exposure with tabular datasets
sbordt.substack.com