🎉 I am extremely excited to announce the biggest news of my career: I will join École Polytechnique @xpolytechnique.bsky.social as an assistant professor in September!
Martin Gubri
@mgubri.bsky.social
Incoming assistant professor at École Polytechnique working on Trustworthy AI Speaking 🇫🇷, English and 🇨🇱 Spanish | he/him https://gubri.eu
Today at #ACL2026, we are presenting out MASEval library for multi-agent system evaluation. @anmolgoel.bsky.social is in San Diego to present poster and live demo! 📍Grand Hall | Session 3: Oral/Posters/Demos B 🕑Sunday 2pm-3.30pm #MultiAgentSystem #AIEvaluation #Python
C Emde@cemde.bsky.social · 5mo ago1/ Evaluating a single agent harness is hard. Evaluating a multi-agent system? Whole different problem. Most eval tools treat the model as the unit of analysis. In multi-agent systems, the system is what matters. That's why we built MASEval 🧵 #AI #Agents #Eval #MultiAgentSystem #LLM
🏆🪩 We just won the best paper award at ICLR'26-CAO with DISCO! I am so proud of my co-authors! Huge congrats to @arubique.bsky.social, Benjamin, and @coallaoh.bsky.social
Alexander Rubinstein@arubique.bsky.social · 11mo ago🪩 Evaluate your LLMs on benchmarks like MMLU at 1% cost. In our new paper, we show that outputs on a small subset of test samples that maximise diversity in model responses are predictive of the full dataset performance. Project page: arubique.github.io/disco-site/ More below 🧵👇
1/ My contract at @parameterlab.bsky.social ended last week, after 2.5 years (since Sept 2023, with some collaboration before). I had the chance to lead research on trustworthy AI for LLMs alongside an incredible group of people. (Neckarfront. All Tübingen researcher have to post it once!)
🎉 Our privacy collapse paper has been accepted at #ACL 2026 (main)! Contextual privacy is fragile: fine-tune an LLM on benign data, and it can overshare personal information. This is silent. Safety suites don't measure contextual privacy, which is a problem now that most applications are agentic.
Anmol Goel@anmolgoel.bsky.social · 7mo ago🚨 Fine-tuning your model to be more helpful or empathetic might be making it less private, without you noticing. In our latest work, we show that benign fine-tuning can silently break contextual privacy in language models while safety & general capabilities appear intact. ⬇️
🌍 We've made LLM watermarking equally robust across all languages we studied, while scaling to 100+ languages! Even sota watermarks can be removed by translating to another language, eg. Tamil. This hits hardest in low-resource languages, where moderation tools are already weak. 🧵
The D&B track now has a larger scope and a new name: Evaluation & Datasets. It focuses on evaluation itself as a scientific object. It is really nice to have somewhere for critical analysis of evaluation and negative results. It was really missing in ML!
NeurIPS Conference@neuripsconf.bsky.social · 5mo agoThe Datasets & Benchmarks track is now "Evaluation and Datasets", with an expanded scope for NeurIPS 2026! Read the call for papers neurips.cc/Conferences/..., and learn more about the changes in our blog post: blog.neurips.cc/2026/03/23/i...
NeurIPS deadline is out! Add the 6th of May to your calendar :)
NeurIPS Conference@neuripsconf.bsky.social · 5mo agoThe NeurIPS 2026 Call for Papers is live: neurips.cc/Conferences/... Abstracts are due May 4, 2026 (AOE), and full papers May 6 (AOE). Please review the handbook neurips.cc/Conferences/... and our Pilot for Strengthening Area Chair Engagement and Calibration blog.neurips.cc/2026/03/23/r...
LLM agents include far more than a model: framework, orchestration, tools, error handling, etc. These harness engineering choices matter, but they're rarely compared. MASEval makes that straightforward. I'm very proud to have supervised its development. Give it a look! ⬇️
C Emde@cemde.bsky.social · 5mo ago1/ Evaluating a single agent harness is hard. Evaluating a multi-agent system? Whole different problem. Most eval tools treat the model as the unit of analysis. In multi-agent systems, the system is what matters. That's why we built MASEval 🧵 #AI #Agents #Eval #MultiAgentSystem #LLM
If you want to get up to speed on what all the benchmarks mean, I wrote a bunch of digests for the popular ones over on the ngrok blog. Designed for people that are interested but not enough to go read all the papers. ngrok.com/blog/ai-benc...
What those AI benchmark numbers mean | ngrok blog
An explanation of 14 benchmarks you're likely to see when new models are released.
ngrok.com
New paper out!🎉 One of our most surprising findings: fine-tuning an LLM on debugging code has unexpected side-effects on contextual privacy. The model learns from printing variables that internal state are ok to share, then generalises this to social situations🤯 A🧵below👇
Anmol Goel@anmolgoel.bsky.social · 7mo ago🚨 Fine-tuning your model to be more helpful or empathetic might be making it less private, without you noticing. In our latest work, we show that benign fine-tuning can silently break contextual privacy in language models while safety & general capabilities appear intact. ⬇️
🎉Thrilled to share that both of my #ICLR2026 submissions were accepted (2/2)! 🪩 DISCO, Efficient Benchmarking: bsky.app/profile/arub... 🩺 Dr.LLM, Dynamic Layer Routing: www.linkedin.com/posts/ahmed-... Huge thanks to my co-authors, especially first authors @arubique.bsky.social & Ahmed Heakl!
Alexander Rubinstein@arubique.bsky.social · 11mo ago🪩 Evaluate your LLMs on benchmarks like MMLU at 1% cost. In our new paper, we show that outputs on a small subset of test samples that maximise diversity in model responses are predictive of the full dataset performance. Project page: arubique.github.io/disco-site/ More below 🧵👇
🧵 Many hidden gems about LLM benchmark contamination in the GAPERON paper! This French-English model paper has some honest findings about how contamination affects benchmarks (and why no one wants to truly decontaminate their training data) Thread 👇
Delighted to announce that 3.5 years after my first first-author paper was accepted at UAI 2022, I've been appointed Area Chair for UAI 2026! 😊 UAI was my first in-person conference right after COVID 1/2
Our #EMNLP2025 paper Leaky Thoughts 🫗 shows that Large Reasoning Models (LRMs) can unintentionally leak sensitive information hidden in their internal thoughts. 📍 Come chat with Tommaso at our poster on Friday 7th, 10:30–12:00 in Hall C3 📄 aclanthology.org/2025.emnlp-m...
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
aclanthology.org
Parameter Lab@parameterlab.bsky.social · last yr.🫗 An LLM's "private" reasoning may leak your sensitive data! 🎉 Excited to share our paper "Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers" was accepted at #EMNLP main! 1/2
🪩 New paper out! Evaluating large models on benchmarks like MMLU is expensive. DISCO cuts costs by up to 99% while still predicting well performance. 🔍 The trick: use a small subset of samples where models disagree the most. These are the most informative. Join the dance party below 👇
Alexander Rubinstein@arubique.bsky.social · 11mo ago🪩 Evaluate your LLMs on benchmarks like MMLU at 1% cost. In our new paper, we show that outputs on a small subset of test samples that maximise diversity in model responses are predictive of the full dataset performance. Project page: arubique.github.io/disco-site/ More below 🧵👇
🎉 Delighted to announce that our 🫗Leaky Thoughts paper about contextual privacy with reasoning models is accepted to #EMNLP main! Huge congrats to the amazing team Tommaso Green, Haritz Puerto @coallaoh.bsky.social @oodgnas.bsky.social
Parameter Lab@parameterlab.bsky.social · last yr.🫗 An LLM's "private" reasoning may leak your sensitive data! 🎉 Excited to share our paper "Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers" was accepted at #EMNLP main! 1/2
Fantastic new paper by @reeserichardson.bsky.social et al. An enormous amount of work showing the extent of coordinated scientific fraud and involvement of some editors. The number of fraudulent publications grows at a rate far outpacing that of legitimate science. www.pnas.org/doi/10.1073/...
PNAS
Proceedings of the National Academy of Sciences (PNAS), a peer reviewed journal of the National Academy of Sciences (NAS) - an authoritative source of high-impact, original research that broadly spans...
pnas.org
Reese Richardson@reeserichardson.bsky.social · last yr.Today, our article "The entities enabling scientific fraud at scale are large, resilient, and growing rapidly" is finally published in PNAS. I hope that it proves to be a wake-up-call for the whole scientific community. reeserichardson.blog/2025/08/04/a...
📢 New paper out: Does SEO work for LLM-based conversational search? We introduce C-SEO Bench, a benchmark to test if conversational SEO methods actually help. Our finding? They don't. But traditional SEO still works because LLMs favour content already ranked higher in the prompt.
Parameter Lab@parameterlab.bsky.social · last yr.🔎Does Conversational SEO actually work? Our new benchmark has an answer! Excited to announce our new paper: C-SEO Bench: Does Conversational SEO Work? 🌐 RTAI: researchtrend.ai/papers/2506.... 📄 Paper: arxiv.org/abs/2506.11097 💻 Code: github.com/parameterlab... 📊 Data: huggingface.co/datasets/par...
Excited to share that our paper "Scaling Up Membership Inference: When and How Attacks Succeed on LLMs" will be presented next week at #NAACL2025! 🖼️ Catch us at Poster Session 8 - APP: NLP Applications 🗓️ May 2, 11:00 AM - 12:30 PM 🗺️ Hall 3 Hope to see you there!
Martin Gubri@mgubri.bsky.social · 2y ago📄 Excited to share our latest paper on the scale required for successful membership inference in LLMs! We investigate a continuum from single sentences to large document collections. Huge thanks to an incredible team: Haritz Puerto, @coallaoh.bsky.social and @oodgnas.bsky.social!
A Bluesky filter to recommend only posts about papers from your followers. This is what I was missing to use Bluesky!
Nikhil Garg@nkgarg.bsky.social · last yr.*Please repost* @sjgreenwood.bsky.social and I just launched a new personalized feed (*please pin*) that we hope will become a "must use" for #academicsky. The feed shows posts about papers filtered by *your* follower network. It's become my default Bluesky experience bsky.app/profile/pape...
I am pleased to announce that our paper on the scale of LLM membership inference from @parameterlab.bsky.social has been accepted for publication at #NAACL2025 as Findings!
Martin Gubri@mgubri.bsky.social · 2y ago📄 Excited to share our latest paper on the scale required for successful membership inference in LLMs! We investigate a continuum from single sentences to large document collections. Huge thanks to an incredible team: Haritz Puerto, @coallaoh.bsky.social and @oodgnas.bsky.social!
🎉We’re pleased to share the release of the models from our Apricot🍑 paper, accepted at ACL 2024! At Parameter Lab, we believe openness and reproducibility are essential for advancing science, and we've put in our best effort to ensure it. 🤗 huggingface.co/collections/... 🧵 bsky.app/profile/dnns...
🍑 Apricot Models - a parameterlab Collection
Fine-tuned models for black-box LLM calibration, trained for "Apricot: Calibrating Large Language Models Using Their Generations Only" (ACL 2024)
huggingface.co
Dennis Ulmer @ACL26@dnnslmr.bsky.social · 2y agoObtaining calibrated confidence scores from LLMs is hard, especially for black-box models. So, can we maybe predict them directly from the generated text? 🤔 Internship work at Parameter Lab with Martin Gubri, Sangdoo Yun, Hwaran Lee, Seong Joon Oh! arxiv.org/abs/2403.059... [1/6]
📄 Excited to share our latest paper on the scale required for successful membership inference in LLMs! We investigate a continuum from single sentences to large document collections. Huge thanks to an incredible team: Haritz Puerto, @coallaoh.bsky.social and @oodgnas.bsky.social!
Parameter Lab@parameterlab.bsky.social · 2y ago🚨📄 Exciting new research! Discover when and at what scale we can detect if specific data was used in training LLMs — a method known as Membership Inference (MIA)! Our findings open new doors for using MIA as potential legal evidence in AI. 🧵 https://arxiv.org/abs/2411.00154
Have a look at the 🍑 Apricot paper that we presented at ACL earlier this year. This project was a wonderful collaboration with @dnnslmr.bsky.social!
Dennis Ulmer @ACL26@dnnslmr.bsky.social · 2y agoObtaining calibrated confidence scores from LLMs is hard, especially for black-box models. So, can we maybe predict them directly from the generated text? 🤔 Internship work at Parameter Lab with Martin Gubri, Sangdoo Yun, Hwaran Lee, Seong Joon Oh! arxiv.org/abs/2403.059... [1/6]
After going to NAACL, ACL and #EMNLP2024 this year, here are a few tips I’ve picked up about attending #NLP conferences. Would love to hear any other tips if you have them! This proved very popular on another (more evil) social media platform, so sharing here also 🙂 My 10 tips:
🌟 Pleased to join Bluesky! As a first post, allow me to share my latest first-author paper, TRAP 🪤, presented at #ACL24 (findings). 🦹💥 We explore how to detect if an LLM was stolen or leaked🤖💥 We showcase how to use adversarial prompt as #fingerprint for #LLM. A thread 🧵 ⬇️⬇️⬇️
