Back from Seoul. My first paper, "An Isotropic Approach to Efficient UQ with Gradient Norms", got a poster and an oral at ProbML, and came away with the Best Paper Award. Still a bit stunned. arxiv.org/abs/2603.29466
Join me and @mariusmosbach.bsky.social to chat about our work on frequency effects in unlearning — and how @ai2.bsky.social's Olmo helped us gain key insights. 💬 AMA: Tue, Oct 28 — 8:00 PT / 16:00 CEST 💡 Bring your questions! 🔗 discord.gg/ai2
Join the Ai2 Discord Server!
The official Discord for Ai2 (The Allen Institute for AI), a nonprofit AI lab. | 2491 members
discord.gg
Ai2@ai2.bsky.social · 10mo agoOur fully open Olmo models enable rigorous, reproducible science—from unlearning to clinical NLP, math learning, & fresher knowledge. Here’s how the research community has leveraged Olmo to make the entire AI ecosystem better + more transparent for all. 🧵
We're presenting “Not all data are unlearned equally” at #COLM2025! We show that data properties shape how LLMs forget — stop by to chat more! 🗓 Wednesday, Oct 8 🕓 4:30–6:30 pm 📍 poster #710 (session 4) paper: arxiv.org/abs/2504.05058 Work with @mariusmosbach.bsky.social @sivareddyg.bsky.social
Not All Data Are Unlearned Equally
Machine unlearning is concerned with the task of removing knowledge learned from particular data points from a trained model. In the context of large language models (LLMs), unlearning has recently re...
arxiv.org
Marius Mosbach@mariusmosbach.bsky.social · last yr.Check out our new paper on unlearning for LLMs 🤖. We show that *not all data are unlearned equally* and argue that future work on LLM unlearning should take properties of the data to be unlearned into account. This work was lead by my intern @a-krishnan.bsky.social 🔗: arxiv.org/abs/2504.05058
very happy to see the trend of a Behind the Scenes section catching on! transparent & honest science 👌 love the detailed montreal spots mentioned consider including such a section in your next appendix! (paper by @a-krishnan.bsky.social arxiv.org/pdf/2504.050...)
Our new paper in #PNAS (bit.ly/4fcWfma) presents a surprising finding—when words change meaning, older speakers rapidly adopt the new usage; inter-generational differences are often minor. w/ Michelle Yang, @sivareddyg.bsky.social , @msonderegger.bsky.social and @dallascard.bsky.social👇(1/12)
📢 #SpeechTech & #SpeechScience researchers! We are thrilled to announce that Prof. Karen Livescu will keynote our Special Session on Interpretable Audio and Speech Models at #Interspeech2025: "What can interpretability do for us (and what can it not)?" 🗓️ Aug 18, 11:00 @interspeech.bsky.social
Announcements
Keynote Speaker Announcement 🔊 30.07.2025 We are delighted to announce the keynote speech t`hat will happen at the special session! Speaker: Prof. Karen Livescu, Toyota Technological Institute at Ch...
sites.google.com
I am excited to announce that my paper "On the reliability of feature attribution methods for speech classification" has been accepted to #Interspeech2025! Co-authors: @hmohebbi.bsky.social, Arianna Bisazza, Afra Alishahi, @grzegorz.chrupala.me Find the preprint here: arxiv.org/abs/2505.16406
On the reliability of feature attribution methods for speech classification
As the capabilities of large-scale pre-trained models evolve, understanding the determinants of their outputs becomes more important. Feature attribution aims to reveal which parts of the input elemen...
arxiv.org
Come to my keynote tomorrow at the first official @queerinai.com workshop at #NAACL2025 to hear about how trans languaging is complex and cool, and how this makes it extra difficult to process computationally. I will have SO many juicy examples!
Chain-of-Thought (CoT) reasoning lets LLMs solve complex tasks, but long CoTs are expensive. How short can they be while still working? Our new ICML paper tackles this foundational question.
Congratulations to Mila members @adadtur.bsky.social , Gaurav Kamath and @sivareddyg.bsky.social for their SAC award at NAACL! Check out Ada's talk in Session I: Oral/Poster 6. Paper: arxiv.org/abs/2502.05670
Incredibly proud of my students @adadtur.bsky.social and Gaurav Kamath for winning a SAC award at #NAACL2025 for their work on assessing how LLMs model constituent shifts.
Language Models Largely Exhibit Human-like Constituent Ordering Preferences
Though English sentences are typically inflexible vis-à-vis word order, constituents often show far more variability in ordering. One prominent theory presents the notion that constituent ordering is ...
arxiv.org
Mila - Institut québécois d'IA@mila-quebec.bsky.social · last yr.Congratulations to Mila members @adadtur.bsky.social , Gaurav Kamath and @sivareddyg.bsky.social for their SAC award at NAACL! Check out Ada's talk in Session I: Oral/Poster 6. Paper: arxiv.org/abs/2502.05670
💡 New ICLR paper! 💡 "On Linear Representations and Pretraining Data Frequency in Language Models": We provide an explanation for when & why linear representations form in large (or small) language models. Led by @jackmerullo.bsky.social, w/ @nlpnoah.bsky.social & @sarah-nlp.bsky.social
DeepSeek-R1 Thoughtology: Let’s <think> about LLM reasoning 142-page report diving into the reasoning chains of R1. It spans 9 unique axes: safety, world modeling, faithfulness, long context, etc. Now on arxiv: arxiv.org/abs/2504.07128
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories We are releasing the first benchmark to evaluate how well automatic evaluators, such as LLM judges, can evaluate web agent trajectories.
Checkout Benno's notes about our impact of interpretability paper 👇. Also, we are organizing a workshop at #ICML2025 which is inspired by some of the questions discussed in the paper: actionable-interpretability.github.io
General Information
ICML 2025 - Vancouver
actionable-interpretability.github.io
Benno Krojer@bennokrojer.bsky.social · last yr.Day 12: From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP arxiv.org/abs/2406.12618 Genuinely one of my favourite papers in recent years! It tries to answer one question that every phd student often asks themselves: Does this research matter?
Check out our new paper on unlearning for LLMs 🤖. We show that *not all data are unlearned equally* and argue that future work on LLM unlearning should take properties of the data to be unlearned into account. This work was lead by my intern @a-krishnan.bsky.social 🔗: arxiv.org/abs/2504.05058
📢Excited to announce our upcoming workshop - Vision Language Models For All: Building Geo-Diverse and Culturally Aware Vision-Language Models (VLMs-4-All) @CVPR 2025! 🌐 sites.google.com/view/vlms4all
Agents like OpenAI Operator can solve complex computer tasks, but what happens when users use them to cause harm, e.g. spread misinformation? To find out, we introduce SafeArena (safearena.github.io), a benchmark to assess the capabilities of web agents to complete harmful web tasks. A thread 👇
📢 #SpeechTech & #SpeechScience researchers! ⏳ Reminder: The #Interspeech2025 deadline is approaching! 🚀 If your work focuses on interpretability in speech & audio, submit through our Special Session and showcase your research! 🎤 #Interpretability @interspeech.bsky.social
Home
Introduction Audio and speech technology has recently achieved unprecedented success in real-world applications, driven primarily by self-supervised pre-training of large neural networks on massive da...
sites.google.com