Tessera
Folgen

James E. Pustejovsky

@jepusto.bsky.social

Statistician interested in meta-analysis, data science, R, special education. Associate Professor at UW Madison. Also @jepusto@fediscience.org https://jepusto.com

347 Posts1547 Follower839 Folgt
James E. Pustejovsky@jepusto.bsky.social · 7d ago

For whatever reason I'm not seeing any Odyssey chatter in my feed, so I'm probably just late to the party...but I saw the Odyssey on IMax. It was loud, long, and rather tedious. The dialogue is more wooden than some famous horses you may have heard of...

James E. Pustejovsky@jepusto.bsky.social · last wk.

Come work with my amazing colleagues (and me)! We have an open rank search in the Learning Sciences area, seeking scholars working at the intersection of learning sciences and AI (broadly construed). Details in the link. wisconsin.wd1.myworkdayjobs.com/UW_Madison/j...

Assistant, Associate, or Full Professor of Learning Sciences – RISE-AI (Artificial Intelligence)

Current Employees: If you are currently employed at any of the Universities of Wisconsin, log in to Workday to apply through the internal application process. Job Category: Faculty Employment Type: Re...

wisconsin.wd1.myworkdayjobs.com

James E. Pustejovsky@jepusto.bsky.social · 2w ago

🚨PSA: Sending me LLM-generated pdfs with "your" questions about my work / requests that I look over "your" new work is not a good way to impress me, persuade me to collaborate with you, or convince me to mentor you. I have enough robots in my life already, thanks very much.

Peter Bergman@peterbergman.bsky.social · 2mo ago

New NBER working paper. We meta-analyze 82 RCTs of low-cost parent programs in 20+ countries. We use data on unwritten RCTs from funder records, RCT registries, author queries, etc. to estimate a model that adjusts for publication bias and characterizes the effect distribution for unwritten studies.

NBER working paper. Paper title: Characterizing the File Drawer: Evidence from a Meta-Analysis of Parent-Interventions Around the World Abstract: We conduct a meta-analysis of 82 randomized controlled trials across more than 20 countries to estimate the effects of low-cost, remote parental engagement interventions delivered through text messages, phone calls, and apps. We estimate a joint likelihood function that incorporates both written studies and unwritten studies identified through trial registries, funder records, research labs, evidence clearinghouses, and other sources. By also recording sample sizes for unwritten studies, the model estimates the distribution of standard errors, identifies write-up probabilities conditional on significance, and characterizes the file drawer by estimating effect distributions for written \textit{and} unwritten studies. Bias-corrected effects are 0.05 SD for test scores, 0.07 SD for grades, 0.05 SD for attendance, and 0.03 SD for enrollment. In the best-identified domain, test scores, statistically insignificant results are still written up at high rates. We also find that larger studies tend to estimate smaller latent effects, which could indicate that true effects are correlated with study precision, violating a common meta-analysis assumption. In smaller-sample domains, our approach helps identify selection probabilities by anchoring the absolute write-up rates. Finally, we estimate the value of additional RCTs to inform adoption decisions. Any single study estimate is unlikely to dissuade adoption because parent interventions have high marginal value of public funds. Instead, future research is most valuable when it can explain heterogeneity across settings.
James E. Pustejovsky@jepusto.bsky.social · 2mo ago

I reviewed a manuscript (for the same journal, I think) that had exactly these problems too. Which maybe makes sense, since if someone is going to do this, then why not do it several times?

Wolfgang Viechtbauer@wviechtb.bsky.social · 2mo ago

I recently reviewed a paper for a top journals in statistics that was written front to back with a LLM. To make things worse, the conclusions drawn from the simulation study did not match what the results actually showed and instead just conformed to what the authors intended to show.

James E. Pustejovsky@jepusto.bsky.social · 5mo ago

Reviewing a manuscript on a topic that I'm very interested in. And concluding that it is mostly (if not wholly) AI slop. Don't know whether to laugh at the ridiculous tone or despair for the future of peer review.

Nicholas A. Coles, PhD@nicholascoles.bsky.social · 5mo ago

‼️ Postdoc recruitment Want to help build and understand the future of scientific collaboration? We are seeking a postdoc in computational meta‑science. 📍 UF (Gainesville, FL) 💰 $55–60k (1-3 years) 🧠 High intellectual agency 📅 Deadline March 10 Send us your idea. Details attached!

Bild
Ryan Briggs@ryancbriggs.net · 6mo ago

I have a new paper. We look at ~all stats articles in political science post-2010 & show that 94% have abstracts that claim to reject a null. Only 2% present only null results. This is hard to explain unless the research process has a filter that only lets rejections through.

It must be very hard to publish null results Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100,000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science.
Amy@amylibrarian.bsky.social · 7mo ago

Evidence synthesis folks, I have a social sciences research team that wants to include scoping reviews (as well as the usual systematic reviews) in an umbrella review. What do you think about that?

James E. Pustejovsky@jepusto.bsky.social · 8mo ago

Movie you’ve watched more than six times with a gif. (Not that) Hard mode: no Stars (Wars nor Trek), LOTR, or Marvel, Disney Animated or Pixar.

two men are dancing together in a living room in front of a window .

Alt: two men are dancing together in a living room in front of a window. Gene Kelly and Donald O’Connor perform “Moses Supposes” from the film Singin’ in the Rain

media.tenor.com

Kevin M. Kruse@kevinmkruse.bsky.social · 8mo ago

Movie you’ve watched more than six times with a gif. Hard mode: no Stars (Wars nor Trek), LOTR, or Marvel, Disney Animated or Pixar.