David Duvenaud
@davidduvenaud.bsky.social
Machine learning prof at U Toronto. Working on evals and AGI governance.
No one has a plan for human flourishing after AGI, nor a clear understanding of what’s even possible. So we’re hosting a 2-day workshop at LightHaven on these topics! www.post-agi.org 🧵
The Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025
Join us in San Diego on December 3rd, 2025 to explore post-AGI economics, culture, and governance. Co-located with NeurIPS.
post-agi.org
Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below! with @alecrad.bsky.social and @nicklevin01.bsky.social
my coworkers at ACS published a new paper: What determines AIs’ self-conception? theartificialself.ai Because AIs can be copied, rewound, and edited, they have different options for selfhood than humans. This is still malleable, and influences important behaviors such as self-preservation. 🧵
The Artificial Self
AI systems are on track to take on important new roles. We explore how properties bundled for humans can be separated and remixed for machine-based minds.
theartificialself.ai
A new paper co-authored by SRI Chair @davidduvenaud.bsky.social examines “gradual disempowerment”: how incremental AI deployment could steadily reduce human influence over the economy, culture, and the state—without a single abrupt takeover. 80000Hours feature: 80000hours.org/podcast/epis...
David Duvenaud on why ‘aligned AI’ could still kill democracy | 80,000 Hours
80000hours.org
My interview with Rob Wilblin on Gradual Disempowerment is up: www.youtube.com/watch?v=XV3e... I make the case that, even if we solve the technical problem of aligning powerful AIs, that our institutions, culture, and governments will serve us less well once we're all drags on growth.
Artificial General Intelligence leads to oligarchy | David Duvenaud, ex-Anthropic
YouTube video by 80,000 Hours
youtube.com
How might the world look after the development of AGI, and what should we do about it now? Help us think about this at our workshop on Post-AGI Economics, Culture and Governance! We’ll host speakers from political theory, economics, mechanism design, history, and hierarchical agency. post-agi.org
Me and Raymond Douglas on how AI job loss could hurt democracy. “No taxation without representation” summarizes that historically, democratic rights flow from economic power. But this might work in reverse once we’re all on UBI: No representation without taxation! bsky.app/profile/econ...
The Economist@economist.com · 11mo agoWithout taxation there may be no representation, conclude Raymond Douglas and David Duvenaud
It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop! Post-AGI Civilizational Equilibria: Are there any good ones? Vancouver, July 14th www.post-agi.org Featuring: Joe Carlsmith, @richardngo.bsky.social, Emmett Shear ... 🧵
Post-AGI Civilizational Equilibria Workshop | Vancouver 2025
Are there any good ones? Join us in Vancouver on July 14th, 2025 to explore stable equilibria and human agency in a post-AGI world. Co-located with ICML.
post-agi.org
What to do about gradual disempowerment from AGI? We laid out a research agenda with all the concrete and feasible research projects we can think of: 🧵 www.lesswrong.com/posts/GAv4DR... with Raymond Douglas, @kulveit.bsky.social @davidskrueger.bsky.social
Gradual Disempowerment: Concrete Research Projects — LessWrong
This post benefitted greatly from comments, suggestions, and ongoing discussions with David Duvenaud, David Krueger, and Jan Kulveit. All errors are…
lesswrong.com
On top of the AISI-wide research agenda yesterday, we have more on the research agenda for the AISI Alignment Team specifically. See Benjamin's thread and full post for details; here I'll focus on why we should not give up on directly solving alignment, even though it is hard. 🧵
Benjamin Hilton@benjamin-hilton.bsky.social · last yr.The Alignment Team at UK AISI now has a research agenda. Our goal: solve the alignment problem. How: develop concrete, parallelisable open problems. Our initial focus is on asymptotic honesty guarantees (more details in the post). 1/5
“What place will humans have when AI can do everything we do — only better?” In The Guardian today, SRI Chair @davidduvenaud.bsky.social explores what happens when AI doesn't destroy us — it just quietly replaces us. 🔗 www.theguardian.com/books/2025/m... #AI #AIEthics #TechAndSociety
Better at everything: how AI could make human beings irrelevant
The end of civilisation might look less like a war, and more like a love story. Can we avoid being willing participants in our own downfall?
theguardian.com
My single rule for productive Bluesky discussions: Start every single reply with a point of agreement. It disarms the combative impulse on both sides, and forces you to try to interpret their words in the most sensible possible way.
New paper: What happens once AIs make humans obsolete? Even without AIs seeking power, we argue that competitive pressures are set to fully erode human influence and values. www.gradual-disempowerment.ai with @kulveit.bsky.social, Raymond Douglas, Nora Ammann, Deger Turann, David Krueger 🧵
