Do LLMs have beliefs, desires, and so on? I think the answer is probably yes. But more than that, the journey leads to fascinating questions how they solve cognitive tasks and what approaches to safety/alignment could work. Here's a first post (of more to come) thinking through these questions ⬇️

About a year ago, I wrote skeptically about LLMs in peer review -- not because of skepticism about their inherent capabilities, but because I don't want the research community to optimize for the taste of any one person/system. What's changed since then?

What’s next for machine learning peer review?

A bit over a year ago, I wrote about the dangers of using LLMs for peer review. The most serious concern I had was algorithmic monoculture: the research community would collectively end up optimizing ...

bryanwilder.substack.com

I'm co-chairing the social impact track at AAAI this year, with Andrew Perrault. Send us your best society-facing work! Personally, I'm especially hoping to see more work speaking to mediators of why and when AI has social impact (or not), like how AI fits into human organizations and decisions.

CMU's Human-AI complementarity workshop is returning this September! Submit abstracts here by July 17; travel funding is available for accepted presenters. www.cmu.edu/ai-sdm/resea...

Deploying algorithmic research in practice is an opaque process. We're organizing a workshop at EC to share behind-the-scenes stories and move the field foward. Call for submissions open! With @nkgarg.bsky.social @ericachiang.bsky.social, Bailey Flanigan sites.google.com/cornell.edu/...

The EAAMO conference deadline is coming up! conference.eaamo.org/cfp/ Great community at intersection of CS-Operations-Econ and social good. Flexible publication format (e.g., non-archival option) and so costless to submit here as well!

LLMs are increasingly used as agents for decisions under uncertainty, e.g. medical diagnosis. But do they act like rational agents with coherent beliefs and preferences? Much of the difficulty is telling whether a model's response to.a prompt ("What is the probability of X?") is a "real" belief.

LLMs are increasingly used as agents for decisions under uncertainty, e.g. medical diagnosis. But do they act like rational agents with coherent beliefs and preferences? Much of the difficulty is telling whether a model's response to.a prompt ("What is the probability of X?") is a "real" belief.

As UKRI explores using LLMs to review grants, it's a good time to revisit Bryan Wilder's excellent blog post. There are a lot of naive reasons to oppose AI review ("you'll never automate human intuition!"). But there are also good reasons, including the *load-bearing role of human disagreement.*

Bryan Wilder@brwilder.bsky.social · last yr.

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

Come talk to me and Angela at NeurIPS on Friday! We argue that "AI for social impact" needs to get more rigorous about evaluating deployments of AI, but also that there are many other forms of impact that get overlooked right now

angela zhou@angelamczhou.bsky.social · 8mo ago

with @brwilder.bsky.social Position Paper: Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards arxiv.org/abs/2510.18238 But we don't know how to do a poster presentation for a position paper 😅

I gave talks at MIT and Harvard this week about "Science with synthetic data". How can generative models help us learn about the actual world (e.g., social systems) in a principled way? Lots of interesting conversations -- more convinced than ever that there's nuanced issues to navigate here.

I’m recruiting students this upcoming cycle at UIUC! I’m excited about Qs on societal impact of AI, especially human-AI collaboration, multi-agent interactions, incentives in data sharing, and AI policy/regulation (all from both a theoretical and applied lens). Apply through CS & select my name!

We're in the process of selecting the location for next year's ACM EAAMO conference! If you're interested in bringing the EAAMO community to your institution, please check out the open call here and get in touch. conference.eaamo.org/call_for_loc...

How can synthetic data from LLMs be used, e.g. for social science, in a principled way? Check out Emily's thread on our NeurIPS paper! Generating paired real-synthetic samples and using both in a method-of-moments framework enables valid inference that benefits when synthetic data is informative.

Emily Byun@yewonbyun.bsky.social · 10mo ago

💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data

Are you a researcher using computational methods to understand cities? @mfranchi.bsky.social @jennahgosciak.bsky.social and I organize an EAAMO Bridges working group on Urban Data Science and we are looking for new members! Fill the interest form on our page: urban-data-science-eaamo.github.io

New piece, out in the Sigecom Exchanges! It's my first solo-author piece, and the closest thing I've written to being my "manifesto." #econsky #ecsky arxiv.org/abs/2507.03600

Submit an abstract to present a poster at EAAMO, deadline July 25! EAAMO is one of my favorite conferences, and a great place for anyone working on ML/algorithms/optimization in social settings. The conference is in Pittsburgh this November. conference.eaamo.org/cfp/call_for...

ACM EAAMO, which is coming to Pitt this Fall, has two events for students: a doctoral consortium and a poster session, both of which are due July 25th - poster session conference.eaamo.org/cfp/call_for... - doctoral consortium conference.eaamo.org/cfp/call_for...

Excited to share that our paper "Learning treatment effects while treating those in need" received the exemplary paper award for AI at EC 2025! This paper grew out collaborations with Allegheny County's human services department and my co-author Pim Welle (at ACDHS). arxiv.org/abs/2407.07596

Learning treatment effects while treating those in need

Many social programs attempt to allocate scarce resources to people with the greatest need. Indeed, public services increasingly use algorithmic risk assessments motivated by this goal. However, targe...

arxiv.org

CMU is hosting a workshop on Human-AI Complementarity for Decision Making this September! Abstract submissions due July 15, travel will be covered for accepted presenters. www.cmu.edu/ai-sdm/resea...

Excited to have this work out at ICML this year! Do LLMs make correlated errors? Yes, and those by the same company, and also more accurate/later generations are more correlated -- increasing algorithmic monoculture arxiv.org/abs/2506.07962

Correlated Errors in Large Language Models

Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully. We conduct a larg...

arxiv.org

Kenny Peng@kennypeng.bsky.social · last yr.

Are LLMs correlated when they make mistakes? In our new ICML paper, we answer this question using responses of >350 LLMs. We find substantial correlation. On one dataset, LLMs agree on the wrong answer ~2x more than they would at random. 🧵(1/7) arxiv.org/abs/2506.07962

Still thinking about this post. The broader point, which should resonate way beyond the specific issue of "peer review," is that human disagreement is not friction and waste. It's a load-bearing, functional part of social and intellectual systems.

Bryan Wilder@brwilder.bsky.social · last yr.

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

Thoughtful take on one aspect of the increasing problem of LLMs leading to “centralization” of thought/writing/etc.

Bryan Wilder@brwilder.bsky.social · last yr.

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

I didn't know about this, but this is objectively procedurally terrible. See Bryan's great analysis 👇 Yes, peer review needs help, but not like this.

Bryan Wilder@brwilder.bsky.social · last yr.

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

Quite the insightful post about the use of LLMs in peer-review. Since that ship has left the stable, let's understand and mitigate the possible adverse effects. bryanwilder.github.io/files/llmrev... E.g., "scientists should demand that any system used as part of peer review be openly accessible"

This is absolutely frightening.

Bryan Wilder@brwilder.bsky.social · last yr.

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research. bryanwilder.github.io/files/llmrev...

Paper: Deep RL + mixed integer programming to plan for restless bandits with combinatorial (NP-hard) constraints. with @brwilder.bsky.social, Elias Khalil, @milindtambe-ai.bsky.social Poster #416 on Friday @ 3–5:30pm bsky.app/profile/lily...

Lily Xu@lilyxu.bsky.social · last yr.

Can we use RL to plan with combinatorial constraints? Our #ICLR2025 paper combines deep RL with mathematical programming to do so! We embed a trained Q-network into a mixed-integer program, into which we can specify NP-hard constraints. w/ brwilder.bsky.social, Elias Khalil, Milind Tambe

Updated abstract deadline is this Thursday, with full paper deadline the following Thursday! Please submit your papers. We will support hybrid presentations for those unable to travel. There is also a non-archival option for those who would like to submit the paper to a journal in the future!

Nikhil Garg@nkgarg.bsky.social · last yr.

🚨 Call for Papers – #EAAMO25 🚨 We invite researchers, practitioners & policymakers to submit work on equity, access, & fairness in algorithms, optimization & mechanism design. 📅 Abstracts due Apr-17 📅 Papers due April-24 🔗 Learn more: conference.eaamo.org/cfp/

🚨 Call for Papers – #EAAMO25 🚨 We invite researchers, practitioners & policymakers to submit work on equity, access, & fairness in algorithms, optimization & mechanism design. 📅 Abstracts due Apr-17 📅 Papers due April-24 🔗 Learn more: conference.eaamo.org/cfp/