And this is why meta analysis is screwed

Alex Holcombe@alexh.bsky.social · 9mo ago

Lisa Bero's first slide at #AIMOS2025 on the evidence base weakening because of commercial influence on research, e.g. false claims of efficacy #metascience

Sorry I gave them so much hell on this one but happy to see some solid discussion about the necessity of qualitative approaches. Bravo.

Samantha Joel@datingdecisions.bsky.social · 11mo ago

We put forth several recommendations to help tackle this problem, including better curation of measures, careful attention to measurement validation (particularly with qualitative approaches), and adoption of SEM approaches that can probe for method effects.

Our lab has three RRs of simulations underway. It’s a shame to tell my students that our top journal will not consider them for publication. Let’s change that!

JK Flake@jkflake.bsky.social · 12mo ago

Help us show there is support for offering registered reports at one of Psychology's leading method's journal! chng.it/TwwnVBScVb

Help us show there is support for offering registered reports at one of Psychology's leading method's journal! chng.it/TwwnVBScVb

Adopt Registered Reports at Psychological Methods

Can you spare a minute to help this campaign?

chng.it

"Preregistration" lumps three distinct practices into one: - Study registration - Protocol - Analysis plans (SAPs) To advance open science, it's critical that we distinguish them. 🌟 🌟 Now in-press paper from Evan Mayo-Wilson, @seangrant.bsky.social, David Moher, me: osf.io/preprints/me...

AMPPS Call for Papers: Replicability and Reproducibility in Methodological Research. Proposals due September 15. @jkflake.bsky.social 

Folks, it's time to submit your papers to PCI Psychology. Here is our invitation to you, taken from our editorial osf.io/preprints/ps...

PCI Psychology@psych.peercommunityin.org · last yr.

PCI Psychology is here!! 🎉🥳 After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org #PsychSciSky #scipub

Check out this awesome new paper about #OpenScience in developing countries! Paper is forthcoming at AMPPS- very proud that we can promote this work. Congrats to the large and international authorship team, and esp @hcp4715.bsky.social for leading the charge. @psychscience.bsky.social

Hu Chuan-Peng@hcp4715.bsky.social · last yr.

🎉 After 2 years of work, our #OpenScience guide for researchers in developing countries is LIVE! Check out our new preprint: osf.io/preprints/ps... We map the opportunities 🌟 + barriers 🧱 facing Global South researchers. *(1/4)*

A new preregistration template for simulation studies is now live on OSF. Thanks to the creators and the preregistration template working group for proposing and shepherding this to make it available to users. See the post if you would like to propose a new template. www.cos.io/blog/introdu...

Introducing the Simulation Studies Preregistration Template: Q&A with Björn S. Siepe, František Bartoš, and Samuel Pawel

Initially submitted through COS’s open call for community-designed preregistration templates, the Simulation Studies Template is now part of the expanding collection of preregistration resources avail...

cos.io

Experimentology is out today!!! A group of us wrote a free online textbook for experimental methods, available at experimentology.io - the idea was to integrate open science into all aspects of the experimental workflow from planning to design, analysis, and writing.

🚨 Calling all dog behavior & cognition researchers - ManyDogs Project is launching ManyDogs 2! 🔍🐕‍🦺We're studying overimitation: Do dogs copy irrelevant actions just b/c their favorite human does them? Email manydogsproject2@gmail.com to collaborate globally and contribute to reproducible science!

I'm very happy to see meta-science on methods research getting more attention in the form of a great new call for papers! I recently gave some talks on the existing literature on the quality and transparency of sim studies & ideas for future improvement. Slides: bsiepe.github.io/talks/2025-0...

JK Flake@jkflake.bsky.social · last yr.

Are you working on issues of replication and reproducibility of methodological research in psychology? Consider submitting a proposal to my call for papers, happy to chat about your ideas or answer any questions! www.psychologicalscience.org/publications...

Are you working on issues of replication and reproducibility of methodological research in psychology? Consider submitting a proposal to my call for papers, happy to chat about your ideas or answer any questions! www.psychologicalscience.org/publications...

🎉 After 2 years of work, our #OpenScience guide for researchers in developing countries is LIVE! Check out our new preprint: osf.io/preprints/ps... We map the opportunities 🌟 + barriers 🧱 facing Global South researchers. *(1/4)*

PCI Psychology is here!! 🎉🥳 After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org #PsychSciSky #scipub

PsyArXivBot@psyarxivbot.bsky.social · last yr.

Peer Community in Psychology: A platform for peer review of preprints across psychology: https://osf.io/m456e

Robustness analysis is going to be a big part of the future of how we think about evidence synthesis and rigor - exciting new journal where you can turn your blog posts of other ways the analyses could have been done into papers! Excited to see some submissions come in!

Journal of Robustness Reports@robustnessreports.bsky.social · last yr.

We are live! Introducing the "Journal of Robustness Reports" – a Diamond Open-Access journal dedicated to publishing short reanalyses of empirical findings. Check out our website and blog post about the journal: 🌐 scipost.org/JRobustRep 📄 www.bayesianspectacles.org/introducing-...

In the workshop we'll follow an example of developing a survey to measuring pickle fanatism for marketing purposes to learn about how to develop and evaluate survey questions for a variety of purposes. The workshop spans qualitative and quantitative methods. Two days of measurement fun!

Andrew F. Hayes@afhayes.bsky.social · last yr.

@jkflake.bsky.social has found a way to merge her passions for science and pickles. Find out how in her course "Scale Development and Psychometrics" offered at Rocky Mountain Methodology Academy in Calgary, July 21-22, 2025. haskayne.ucalgary.ca/ccram/academy A trip to Banff is included.

HELP: Many of you will be familiar with the dataset that is often used to teach multilevel modeling with variable such as "MATHACH" and "SES" -- these data are used in papers, textbooks, and software packages. You may have the dataset on your computer, BUT, is it public use data?

Chuffed and excited to announce a new journal section in Cortex: *Methods and Assumptions* is open for submissions starting today! Read the opening editorial here (OA): www.sciencedirect.com/science/arti... 1/

Methods and Assumptions: A new section in Cortex

sciencedirect.com

Attention metascience & AI enthusiasts: SSHRC is offering $140K/2yr postdoc fellowships, and I'd be happy to sponsor someone to work in my lab at UofT. If you're interested in how AI might transform scientific practice (for better or worse), apply by April 8. Let's question everything together.

I have to fix it in my head. This survey validates me.

Julia M. Rohrer@dingdingpeng.the100.ci · last yr.

I asked DeepSeek to develop a scale to measure addiction to addiction scale development and of course it delivered.

I’ve long used FiveThirtyEight’s interactive “Hack Your Way To Scientific Glory” to illustrate the idea of p-hacking when I teach statistics. But ABC/Disney killed the site earlier this month :( So I made my own with #rstats and Observable and #QuartoPub ! stats.andrewheiss.com/hack-your-way/

Hope to see more papers like this that question the performance and trade-offs of more complicated and newer methods with a sample of datasets (not just simulations that can benefit from cherry picking). Combats the publication bias towards new methods and demonstrates on-going comparative work.

Gaël Varoquaux@gaelvaroquaux.bsky.social · last yr.

🎓Paper time!✨ #ICLR spotlight. Concluding of 5 years of research on missing values handling for prediction: Beware of diminishing returns in imputation for prediction. 1/8

I'm hiring! I'm looking for 1️⃣ pre-doc and 2️⃣ post-docs. If you're fascinated by media effects, quantitative methods, and open science, please consider applying 🙂 International applications welcome! Application Pre-doc: jobs.uzh.ch/job-vacancie... Application Post-doc: jobs.uzh.ch/job-vacancie...

👀

Nick Huntington-Klein@nickchk.com · last yr.

After a long wait, the working paper for the Many-Economists Project: The Sources of Researcher Variation in Economics. We had 146 teams perform the same research three times, each time with less freedom. What source of freedom leads to different choices and results? papers.ssrn.com/sol3/papers....

HelioCampus is hiring! We have openings for data scientists and data analysts. www.heliocampus.com/company/care... I'm the hiring manager for the data scientist role, and we're specifically looking for someone with experience working with student data in the higher ed industry.

Careers | HelioCampus

Explore our career opportunities and join our diverse team of higher ed experts. Read more.

heliocampus.com

The methods and statistics group at @tilburg-university.bsky.social get it.

The looks great: “cross-cultural developmental research poses distinct problems for ensuring high construct validity, owing to the nuances of working with children; the standard approach of transporting protocols designed & validated in one population to another risks low construct validity”