A many-designs study of 516 algorithms challenges the predictability of wisdom of the crowd forecast aggregation: WoCCAP consortium

AUTOMATING COMPUTATIONAL REPRODUCIBILITY My colleague @philipjakobbln.bsky.social and I are currently engaged in a research project where we reproduce scientific results en masse. To that end, we built rigor.me, a platform for automatically reproducing papers using agents 🧵

Americans say they want less inequality. Why don’t they support policies to reduce it? New paper: across 31 nationally representative survey waves (N=384,248) and a preregistered experiment (N=1,009), Americans favor PREdistributive over REdistributive policies 🧵 osf.io/preprints/so...

🚨New Paper in PNAS: "Refugee Labor Market Integration at Scale: Evidence from Germany’s Fast-Track Employment Program" www.pnas.org/doi/10.1073/... Ungated preprint osf.io/preprints/socarxiv/px9ew_v3 w/ J Hainmueller, D Hangartner, @niklas-harder.bsky.social & E Vallizadeh #econtwitter #econsky

Refugee labor market integration at scale: Evidence from Germany’s fast-track employment program | PNAS

Governments face persistent challenges in integrating refugees into the local labor market, and many past interventions have shown limited impact. ...

pnas.org

First day of my research visit at Oxford! Many thanks to @zparolin.bsky.social for the generous invitation. I’ll be presenting on simulating human behavior with LLMs on Jan 28 at the @nuffieldcollege.bsky.social Sociology Seminar. If you’ll be around Oxford, feel free to reach out!

Effect sizes in social science survey experiments are typically small, requiring large samples and budgets for sufficient statistical power. It gets even trickier because increasing n/$ yields only diminishing returns to statistical power. This study points to an important barrier to credibility👇 1/

Public Opinion Quarterly - POQ@poqjournal.bsky.social · 8mo ago

Survey experiments have become a popular methodology among social scientists. Has it been effective? In POQ, Rauf et al. study the efficacy of 100 survey experiments. Their results show that a majority of hypotheses were not supported. Read now: doi.org/10.1093/poq/...

To simulate-then-validate, to validate-then-simulate, or to calibrate: that is the question. We discuss ways to simulate human responses to behavioral science experiments using LLMs and strategies to address their limitations. Check out our preprint!

Jessica Hullman@jessicahullman.bsky.social · 8mo ago

Many think LLM-simulated participants can transform behavioral science. But there's been a lack of accessible discussion of what it means to validate LLMs for behavioral scientists. Under what conditions can we trust LLMs to learn about human parameters? Our paper maps the validation landscape. 1/

I'm super happy & proud to be able to work with this rock star group of a team (and academic guests)! 🧠🧠🧠💫🙏 @goetheuni.bsky.social @infer-frankfurt.bsky.social - and it's really nice to check out parts of Frankfurt that I hadn't been to as part of our team event

Check out this article on leveraging AI for conducting social science research!

Stanford HAI@stanfordhai.bsky.social · last yr.

Social science research can be time-consuming, expensive, and hard to replicate. But with AI, scientists can now simulate human data and run studies at scale. Does it actually work? hai.stanford.edu/news/social-...

Excited to continue learning about the latest #CSS at @ic2s2.bsky.social! I’ll be at the Social Prediction Session, presenting the mixed subjects design on combining human and LLM data in experiments. Paper with Michael Howes and @austin-van-loon.bsky.social. Come join us! doi.org/10.1177/0049...

New in Sociological Methods & Research: Soc PhD candidate @davidbroska.bsky.social, @austin-van-loon.bsky.social, & Michael Howes show how combining human subjects and large language models can yield precise estimates at low cost, with implications for scientific productivity doi.org/10.1177/0049...

How can we leverage generative AI to advance social science methods and research? Daniel Karell and Thomas Davidson led a special issue in Sociological Methods & Research to find out. Special kudos to them! journals.sagepub.com/doi/10.1177/...

Mixed feelings about silicon subjects (LLM predictions of human behavior) as replacements for human subjects? Consider the mixed subjects design. 🚨Now published at Sociological Methods and Research🚨 doi.org/10.1177/0049...

Happy to share my new paper with Cat Dang Ton and @eollion.bsky.social on how to use generative LLMs for extracting information from textual data (conditionally accepted at Sociological Methods & Research) Here's a rundown.. osf.io/preprints/so...

🚨 ACCEPTED AT SMR 🚨
Confused by colleagues who seem to want to study LLMs instead of humans? Frustrated by skeptics (e.g., myself 8 months ago) who dismiss LLMs as a potential source of data on human behavior? Check out our paper for a new way forward: osf.io/j3bnt_v3/