This morning at #ic2s2 I have the chance to present ongoing work on applying a SEM from survey methods to LLM text annotations: 👉 Evaluating LLM Text Annotations Without Ground Truth 📍10:45, Mansfield (210), Understanding LLMs w/ @maximiliankreutner.bsky.social Alex Cernat & @mstrohm.bsky.social

@milanlp.bsky.social x #WomeninAI

For today's reading group, @marlutz.bsky.social presented "State media control influences large language models" by Waight et al. (2026) Paper: www.nature.com/articles/s41... #NLProc

We are delighted to welcome @marlutz.bsky.social to our lab over the next few months! 🎉 She'll work on the representation of different demographic groups in LLMs. #NLProc

🚨 TADA Speaker Series Spring 2026 schedule is here! 🚨 We've assembled a fantastic lineup of researchers exploring the future of survey research in the age of LLMs. Mar 18 - May 27, online at 17:00 CEST. Join us! More info & signup: tada.cool

Survey-style tests developed for humans may not predict how LLMs actually behave. Our #EACL2026 paper shows they can even be misleading when measuring racism and sexism! Check out the paper 👇🏼

Jana Jung@janajung.bsky.social · 5mo ago

Are you using survey-style questionnaires designed for humans to measure characteristics of LLMs? In our #EACL2026 paper, we evaluate both the reliability and validity of such tests and found that their scores do not reflect real-world model behavior. In fact, they can be deceptive! 🧵1/3

Very honored to be one out of seven outstanding papers at this years' EMNLP :) Huge thanks to my amazing collaborators @fatemehc.bsky.social @anamarasovic.bsky.social @boknilev.bsky.social , this would not have been possible without them!

👋🏼 I'm at #EMNLP2025 presenting "The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for LLMs" 🕑 Thu. Nov 6, 12:30 - 13:30 📍 Findings Session 2, Hall C3

Marlene Lutz@marlutz.bsky.social · 9mo ago

🚨New paper alert🚨 🤔 Ever wondered how the way you write a persona prompt affects how well an LLM simulates people? In our #EMNLP2025 paper, we find that using interview-style persona prompts makes LLM social simulations less biased and more aligned with human opinions. 🧵1/7

🚨New paper alert🚨 🤔 Ever wondered how the way you write a persona prompt affects how well an LLM simulates people? In our #EMNLP2025 paper, we find that using interview-style persona prompts makes LLM social simulations less biased and more aligned with human opinions. 🧵1/7

👋 #ACL2025NLP 🇦🇹 @marlutz.bsky.social and I are presenting our poster on demographic representativeness of LLMs today! 🕦 10:30-12:00 📍 Hall X5 (board 1 or 14 according to different sources 🧐) Here’s the paper on ACL anthology: aclanthology.org/2025.finding... Drop by!

Missing the Margins: A Systematic Literature Review on the Demographic Representativeness of LLMs

Indira Sen, Marlene Lutz, Elisa Rogers, David Garcia, Markus Strohmaier. Findings of the Association for Computational Linguistics: ACL 2025. 2025.

aclanthology.org

Indira Sen@indiiigo.bsky.social · last yr.

Do LLMs represent the people they're supposed simulate or provide personalized assistance to? We review the current literature in our #ACL2025 Findings paper and investigating what researchers conclude about the demographic representativeness of LLMs: osf.io/preprints/so... 1/

Joint work w/ @marlutz.bsky.social, Elisa Rogers, @dgarcia.eu and @mstrohm.bsky.social You can find our code and annotated dataset of papers here: github.com/Indiiigo/LLM... We annotated way more things, e.g., LLM used, response format, so please check it out! 5/5

GitHub - Indiiigo/LLM_rep_review: Systematic Review of the Demographic Representativeness of LLMs

Systematic Review of the Demographic Representativeness of LLMs - Indiiigo/LLM_rep_review

github.com