Fabian Schaipp
@fschaipp.bsky.social
Researcher in Optimization for ML at Inria Paris. Previously at TU Munich. https://fabian-sp.github.io/
🚋 New blog post: On "infinite" learning-rate schedules and how to construct them from one checkpoint to the next. fabian-sp.github.io/posts/2025/0...
Infinite Schedules and the Benefits of Lookahead
TL;DR: Knowing the next training checkpoint in advance (“lookahead”) helps to set the learning rate. In the limit, the classical square-root schedule appears on the horizon.
fabian-sp.github.io
Learning rate schedules seem mysterious? Why is the loss going down so fast during cooldown? Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization. A short thread on our latest paper 🚞 arxiv.org/abs/2501.18965
The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedul...
arxiv.org
That time of the year again, where you delete a word and latex manages to make the line <longer>.
Want all NeurIPS/ICML/ICLR papers in one single .bib file? Here you go! 🗞️ short blog post: fabian-sp.github.io/posts/2024/1... 📇 bib files: github.com/fabian-sp/ml-bib
A Bibliography Database for Machine Learning
Getting the correct bibtex entry for a conference paper (e.g. published at NeurIPS, ICML, ICLR) is annoyingly hard: if you search for the title, you will often find a link to arxiv or to the pdf file,...
fabian-sp.github.io
Generating cat videos is nice, but what if you could tackle real scientific problems with the same methods? 🧪🌌 Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology. 🐙: github.com/PolymathicAI... 📜: openreview.net/pdf?id=00Sx5...
Not so fun exercise: take a recent paper that you consider exceptionally good, and one that you think is mediocre (at best). Then look up their reviews on ICLR 2025. I find these reviews completely arbitrary most of the times.
my French 🇨🇵 digital bank (supposedly!) today asked me (via letter) to confirm an account action via sending them a signed letter. wtf
I made a #starterpack for computational math 💻🧮 so please 1. share 2. let me know if you want to be on the list! (I have many new followers which I do not know well yet, so I'm sorry if you follow me and are not on here, but want to - drop me a note and I'll add you!) go.bsky.app/DXdZkzV