(New Release!) How are people really using AI? Today, researchers at MIT, Stanford, + 12 academic institutions are launching the AI Observatory: public infra for measuring how people actually use AI assistants in the wild. 🔗 ai-observatory.org 📜 www.dataprovenance.org/ai_observato...

I wrote about how, beyond what AI companies tell us, we still know very about real people's AI use--and the AI Observatory project from @shaynelongpre.bsky.social @ankareuel.bsky.social + team trying to change this: www.technologyreview.com/2026/08/18/1...

We still don’t know how people are really using AI

But a new study shows that work use cases make up less of the picture than AI companies claim.

technologyreview.com

1/2 We've heard of the curse of dimensionality. But what about the curse of multilinguality? "If you try to pack more languages into the same size model, they are each going to degrade," said @shaynelongpre.bsky.social of @mit.edu at the Simons Institute.

Excited to publish this piece!

Justin Hendrix@justinhendrix.bsky.social · 10mo ago

Policymakers must recognize the open source AI ecosystem is where influence is being negotiated: not just which models exist, but which are used; not just who can train a trillion-parameter network, but who can make it deployable, modifiable, and relevant, say Lucie-Aimée Kaffee and Shayne Longpre.

I’ll be hanging out at our poster on membership inference, but in the same slot Brian Lester will present our work on “The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text” (poster 102)! [https://arxiv.org/abs/2506.05209]

A. Feder Cooper@afedercooper.bsky.social · 10mo ago

[NeurIPS '25] Really excited to present “Exploring the limits of strong membership inference attacks on large language models” (poster 1300) this morning (Friday December 5, 11am-2pm in Exhibit Hall C-E)! [https://arxiv.org/abs/2505.18773]

"China has overtaken the US in the global market for 'open' artificial intelligence models, gaining a crucial edge over how the powerful technology is used around the world."

China leapfrogs US in global market for ‘open’ AI models

Beijing-backed technology gains ground as American giants hold fast to ‘closed’ AI strategies

ft.com

Who is winning the open AI race? Our new study Economies of Open Intelligence maps @hf.co 851k models' downloads 2020→2025. 1) Power rebalance: US tech ↓; China + community ↑ 2) Models size & efficient ↑ (MoE, quant, multimodal) 3) Intermediary layers ↑ (adapters/quantizers) 4) Transparency ↓ /🧵

This is some legit really impressive work!!

Shayne Longpre@shaynelongpre.bsky.social · 11mo ago

📢Thrilled to introduce ATLAS 🗺️: the largest multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍 Is scaling diff by lang? 🧙‍♂️ Can we model the curse of multilinguality? ⚖️ Pretrain vs finetune from checkpoint? 🔀 X-lingual transfer scores across langs? 1/🧵

📢Thrilled to introduce ATLAS 🗺️: the largest multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍 Is scaling diff by lang? 🧙‍♂️ Can we model the curse of multilinguality? ⚖️ Pretrain vs finetune from checkpoint? 🔀 X-lingual transfer scores across langs? 1/🧵

Which, whose, and how much knowledge do LLMs represent? I'm excited to share our preprint answering these questions: "Epistemic Diversity and Knowledge Collapse in Large Language Models" 📄Paper: arxiv.org/pdf/2510.04226 💻Code: github.com/dwright37/ll... 1/10

Delighted to see BigGen Bench paper receive the 🏆best paper award 🏆at NAACL 2025! BigGen Bench introduces fine-grained, scalable, & human-aligned evaluations: 📈 77 hard, diverse tasks 🛠️ 765 exs w/ ex-specific rubrics 📋 More human-aligned than previous rubrics 🌍 10 languages, by native speakers 1/

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

How should regulatory proposals adapt to the prevalence of general-purpose AI when the global geopolitical order is being reconfigured? @atoosakz.bsky.social, Deirdre K. Mulligan, @randomwalker.bsky.social, @alondra.bsky.social, & @shaynelongpre.bsky.social weigh in: youtu.be/cRsbjGFPJaM?...

Day 1 Opening Remarks and Panel 1: Regulating AI in Democratic Upheaval (AI & Democratic Freedoms)

YouTube video by Knight First Amendment Institute

youtu.be

🛬 in Singapore for #ICLR2025! DM me to catch up—but only if you have a local food/bar/event rec!

Thrilled our global data ecosystem audit was accepted to #ICLR2025! Empirically, it shows: 1️⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024). 2️⃣ YouTube is now 70%+ of speech/video data but could block third-party collection. 3️⃣ <0.2% of data from Africa/South America. 1/

📍EVENT: Day 2 of our “AI and Democracy” symposium will be kicking off shortly. Programming will begin with welcome remarks from George Deodatis @columbiaseas.bsky.social at 9:30am ET. #AIDemocraticFreedoms Watch the full event on our livestream here: www.youtube.com/watch?v=X1gj...

Artificial Intelligence and Democratic Freedoms (Day 2)

YouTube video by Knight First Amendment Institute

youtube.com

Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative! 🌍 18 languages (high-, mid-, low-) 📚 21k questions (55% require image understanding) 🧪 STEM, social science, reasoning, and practical skills

Panel 1: Regulating AI in a Time of Democratic Upheaval starts in approximately 5 minutes. Panelists: @atoosakz.bsky.social, @randomwalker.bsky.social, @alondra.bsky.social, and Deirdre K. Mulligan. Moderator: @shaynelongpre.bsky.social. #AIDemocraticFreedoms

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

Excited to speak at the workshop on Technical AI Governance in Vancouver this summer! #ICML2025

Technical AI Governance @ ICML 2025@taig-icml.bsky.social · last yr.

📣We’re thrilled to announce the first workshop on Technical AI Governance (TAIG) at #ICML2025 this July in Vancouver! Join us (& this stellar list of speakers) in bringing together technical & policy experts to shape the future of AI governance! www.taig-icml.com

#AI is evolving fast, and so are its flaws. A fresh approach to finding and reporting AI bugs is long overdue. Great initiative by @shaynelongpre.bsky.social and team, transparency and accountability in AI development are essential! #AISafety #ResponsibleAI #AIEthics #MIT

Researchers Propose a Better Way to Report Dangerous AI Flaws

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

wired.com

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

Researchers Propose a Better Way to Report Dangerous AI Flaws

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

wrd.cm

Thank you @willknight.bsky.social for excellent coverage of our new proposal! www.wired.com/story/ai-res...

Researchers Propose a Better Way to Report Dangerous AI Flaws

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

wired.com

Shayne Longpre@shaynelongpre.bsky.social · 2y ago

What are 3 concrete steps that can improve AI safety in 2025? 🤖⚠️ Our new paper, “In House Evaluation is Not Enough” has 3 calls-to-actions to empower evaluators: 1️⃣ Standardized AI flaw reports 2️⃣ AI flaw disclosure programs + safe harbors. 3️⃣ A coordination center for transferable AI flaws. 1/🧵

What are 3 concrete steps that can improve AI safety in 2025? 🤖⚠️ Our new paper, “In House Evaluation is Not Enough” has 3 calls-to-actions to empower evaluators: 1️⃣ Standardized AI flaw reports 2️⃣ AI flaw disclosure programs + safe harbors. 3️⃣ A coordination center for transferable AI flaws. 1/🧵

Very glad to join this paper organized by @shaynelongpre.bsky.social. Here's the paper itself: crfm.stanford.edu/2025/03/13/t...

Researchers Propose a Better Way to Report Dangerous AI Flaws

After identifying major flaws in popular AI models, researchers are pushing for a new system to identify and report bugs.

wired.com

Bringing transparency to the data used to train artificial intelligence mitsloan.mit.edu/ideas-made-t...

Bringing transparency to the data used to train artificial intelligence | MIT Sloan

Using the wrong datasets to train AI models can result in legal risks, bias, or lower-quality models. The Data Provenance Initiative’s tool can help.

mitsloan.mit.edu

Thrilled to be at #AAAI2025 for our tutorial, “AI Data Transparency: The Past, Present, and Beyond.” We’re presenting the state of transparency, tooling, and policy, from the Foundation Model Transparency Index, Factsheets, the the EU AI Act to new frameworks like @MLCommons’ Croissant. 1/

Really excellent explainer by @shaynelongpre.bsky.social‬ that clearly lays out what's at stake in the "AI crawler wars"

Shayne Longpre@shaynelongpre.bsky.social · 2y ago

I wrote a spicy piece on "AI crawler wars"🐞 in @technologyreview.com (my first op-ed)! While we’re busy watching copyright lawsuits & the EU AI Act, there’s a quieter battle over data access that affects websites, everyday users, and the open web. 🔗 www.technologyreview.com/2025/02/11/1... 1/

Re: the FTC and the platforms We *should* be concerned about platform power over speech, but it isn’t censorship. As the Supreme Court said last year, the companies’ editorial decisions to moderate content are protected by the First Amendment. 1/

Let’s see if the algorithm and data remains transparent.