Great chapter on a very important topic. Construct validity continues to be one of the biggest challenges in research characterizing the behavior of LLMs as well.
Mike Frank@mcxfrank.bsky.social · 4w agoIf you don't know how reliable your measure is, you're wasting your participants' time. Ch 8 of Experimentology argues that measurement reliability and validity deserve more attention than most experimentalists give them. 🧵 experimentology.io
Will be presenting this work measuring behavioral sensitivity to false belief manipulations in a sample of open-weight LMs at tomorrow's poster session (11am). Stop by if you're interested! #ACL2026 aclanthology.org/2026.acl-lon...
Most of the statistical tests you learned in stats class — t-test, ANOVA, correlation — are actually special cases of a single thing: linear regression. Ch 7 of Experimentology argues that thinking in models, not tests, is more flexible and a better foundation for theory. 🧵 experimentology.io
Seems like a good time to share our new preprint about model openness! (with @catherinearnett.bsky.social @tylerachang.bsky.social Pamela D. Rivière, Samuel M. Taylor @camrobjones.bsky.social @seantrott.bsky.social @rplevy.bsky.social Ben Bergen, and Micah Altman): arxiv.org/abs/2603.26539
In film, "we'll fix it in post" is what you say when something went wrong on set and you don't want to redo it. AI research has made it our entire methodology: train the model, then patch whatever comes out. Our new ICML oral argues this can't be the basis of a science of AI. 🧵
If you're a faculty, research scientist, postdoc, or senior PhD in any area of science, select your top 6 virtues at this link: shorturl.at/bV2lF You can also tell us if you think we missed any. We want broad participation, so pls RT! 🙏 In collab w/@devezer.bsky.social @statmodeling.bsky.social
Epistemic Virtues for Scientific Inquiry — Community Survey
In Six Memos for the Next Millenium, Italo Calvino proposed six literary virtues he felt should be preserved and cultivated in literature regardless of how the world changed: lightness, quickness, exa...
shorturl.at
Very excited to announce that I'll be starting as an Assistant Professor in the Psychology department at Rutgers University-Newark in January 2026!
Will be presenting a new paper on generalizability in mechinterp research at the 2025 NeurIPS MechInterp workshop! Thread below. #NeurIPS
Does vision training change how language is represented and used in meaningful ways?🤔The answer is a nuanced yes! Comparing VLM-LM minimal pairs, we find that while the taxonomic organization of the lexicon is similar, VLMs are better at _deploying_ this knowledge. [1/9]
I think understanding which factors lead to convergence and divergence (both in behavior and internal mechanisms) across networks is crucial to understanding what kinds of systems we're studying and what kinds of claims we can generalize across model instances. Very cool work!
Ann Huang@annhuang42.bsky.social · 8mo ago📍Excited to share that our paper was selected as a Spotlight at #NeurIPS2025! arxiv.org/pdf/2410.03972 It started from a question I kept running into: When do RNNs trained on the same task converge/diverge in their solutions? 🧵⬇️
📍Excited to share that our paper was selected as a Spotlight at #NeurIPS2025! arxiv.org/pdf/2410.03972 It started from a question I kept running into: When do RNNs trained on the same task converge/diverge in their solutions? 🧵⬇️
A confounding thing for the linguistics of LMs: the best way to assess their grammatical ability is string probability. Yet string probability and grammaticality are famously not the same! Really excited to have this out, where we give a formal account, w/ experiments, of how to make sense of that!
Jennifer Hu@jennhu.bsky.social · 9mo agoNew work to appear @ TACL! Language models (LMs) are remarkably good at generating novel well-formed sentences, leading to claims that they have mastered grammar. Yet they often assign higher probability to ungrammatical strings than to grammatical strings. How can both things be true? 🧵👇
Hard to process the news about Harvard and international students. Other universities should stand in solidarity with our colleagues who are being persecuted.
Please read my essay in TIME, which @science.org did not do carefully before publishing this assertion. time.com/7285045/resi...
Why I’m Resigning from the NSF and Library of Congress
I cannot participate in systems that require dishonesty as the price of belonging.
time.com
My awesome colleague @seantrott.bsky.social quoted on understanding the nature of our new robot overlords in @nytimes.com www.nytimes.com/2024/12/16/o...
Opinion | How Does A.I. Think? Here’s One Theory.
It may employ a kind of reasoning humans use all the time — but you’ve probably never heard of it.
nytimes.com