Ah, I never used it. I had never been deeply interested in 1-shot reproducibility of full papers until realizing it made revising so much easier; every table and number is just a reference to artifact generated by the code. Data can change, errors can be found; just rerun it.

⚙️ New on the COS blog: Principal Research Scientist Noah Haber (@whaleactually.com) shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-source R package. www.cos.io/blog/toward-...

Toward Collaborative Reproducible Manuscripts

COS Principal Research Scientist Noah Haber shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-sour...

cos.io

For most scientific papers, coathors collaborate in the manuscript doc and someone manually enters analysis results/tables/figures into it. In this case, reproducibility effectively stops at the data and code. What this blog post presupposes is, what if it didn't? www.cos.io/blog/toward-...

Toward Collaborative Reproducible Manuscripts

The first active phase of COS's Benchmarking LLM Agents on Scientific Tasks project has produced ReplicatorBench—a benchmark for evaluating LLM agents on research replication in social & behavioral sc...

cos.io

When I started working in global health a little over 10 years ago, infectious disease (particularly policy) was said to be a declining field to be almost entirely replaced with chronic disease. That ... did not happen.

Had a small part of this that I am pretty proud of: Full push-button reproducible manuscripts from data to manuscript. Nearly every stat, figure, number, table, etc is pulled directly from analysis code and data and "knit" with manuscript text written in Google Docs. Blog post coming soon ...

Center for Open Science@cos.io · 4mo ago

Today, the SCORE program releases its primary results! 865 researchers examined research credibility across the social and behavioral sciences, publishing three papers in Nature + five preprints. 📑 Explore the papers: www.nature.com/colle... ℹ️ Read more about SCORE: www.cos.io/score

"Elapsed time" is to Intention to Treat as "Moving time" is to Per-Protocol. I will not explain any further.

A few of the first recruitment attempts to US-based journals have been met with a "times are too uncertain to take on a project like this," as expected Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.

Noah Haber@whaleactually.com · last yr.

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial. This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs. Now piloted and ready for the main stage, and looking for partners! 🧵👇

The "meta trial" idea is maybe the most ambitious thing I've ever gotten to try in real life. In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale. Not just for journal policy. For policy period.

Noah Haber@whaleactually.com · last yr.

What we would really want in the end is a BUNCH of compatible, but realistic and pragmatic trials of custom variations on this policy and get to wrap it all up in a nice meta analysis. For the last year or so, we've been piloting a pretty wild approach to getting exactly that: A study-in-a-kit

This meta trial idea is pretty cool. Eventually you could arrive at a situation where journals help researchers get individual credit for research labor on large collaborative projects

Noah Haber@whaleactually.com · last yr.

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial. This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs. Now piloted and ready for the main stage, and looking for partners! 🧵👇

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial. This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs. Now piloted and ready for the main stage, and looking for partners! 🧵👇

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial. This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs. Now piloted and ready for the main stage, and looking for partners! 🧵👇

death to "is it A or is it B that causes Y" long live "how much do A and B contribute to Y"

Triangle folks: I've got water, a bunch of 5 gallon jugs, tools galore, and plenty of time to deliver food/water/whatever to where it needs to go. Throw a DM my way if you need anything at all.

Welp, it finally happened: I messed up a thing in a study involving participants, caught it, and am working on submitted a complete report and mitigation plan of the issue to the IRB. Fortunately the damage to my pride is almost certainly larger than any risk to participants. But still. Humbling.

o hai, i am in copenhagen, sciencing yur sciences

Noah Haber@whaleactually.com · last yr.

Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos. If you're around, come hang out!

Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos. If you're around, come hang out!

RCT-only plus liberal use of guidelines for power, measurement error, heterogeneous treatment effects, examining attrition, etc. We like to set up methods as the gold standard but then put little back doors in rather than own up to our true uncertainty.

Remember folks: the proverbial "gold standard" for evidence was driven by our own mainstream institutional evidence based medicine movement. A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else). What could possibly go wrong?

Added my name. There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends. Right now, it is time to make yourself heard, however you choose to do it. actionnetwork.org/petitions/op...

Sign The Open Letter to Stand Up For Science Now!

Science is under siege. Trump’s latest Executive Order calls for politically appointed science commissars to evaluate research. Join us in adding your name to our open letter condemning Trump’s escala...

actionnetwork.org

Stand Up for Science!@standupforscience.net · last yr.

🚨 URGENT RESPONSE CALL 🚨 Trump’s Fool’s Gold Science EO is a dangerous sham. It gives his appointees the power to dismiss entire bodies of research and punish researchers who fail to fall in line with his agenda. We’ve launched an open letter. SIGN & SHARE NOW: actionnetwork.org/petitions/op...

Coming soon: the most fully realized preregistration/registered report ever (I think) * Pre-written methods and results section * Full reproducible data -> manuscript text and fig pipeline * Pre-populated data w/ full trial(s) simulation * Multiple levels (trials + meta analysis) for all

how it started / how it's going www.science.org/doi/10.1126/... www.aaup.org/news/cowardi...

We are experiencing an assault on science unparalleled by anything I’ve seen in my life. It’s not one issue or another anymore, the entire institution is under attack by the most powerful individuals in the country. This Friday, where will you be? standupforscience2025.org

STAND UP FOR SCIENCE

March 7, 2025. Washington DC and nationwide. Because science is for everyone.

standupforscience2025.org

Stand Up for Science!@standupforscience.net · last yr.

Welcome to the Bluesky account for Stand Up for Science 2025! Keep an eye on this space for updates, event information, and ways to get involved. We can't wait to see everyone #standupforscience2025 on March 7th, both in DC and locations nationwide! #scienceforall #sciencenotsilence

These two takes are VERY different. Both identify the severity of the threat we face. But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.

Magdalena Skipper@magdalenaskipper.bsky.social · last yr.

Great minds… within 24 hrs both us at @nature.com and @science.org make similar appeals to the research community. And @holdenthorp.bsky.social and I didn’t even exchange notes about it ;-) 🧪 www.nature.com/articles/d41... www.science.org/doi/10.1126/...

Good news: Added a new use on a slide for how preregistration might enhance research, yay! Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.

The practice of science is rejecting and challenging authority, not being obedient to it. Science is punk.

The problem is that some data always lie while some always tell the truth

Absolutely not. This time calls for an array of approaches, righteous (and just plain right) fury included. This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful. www.science.org/doi/10.1126/...

This is fair, but remember if this is something new to you, or just becoming apparent, there's a wide body of research, activism, resistance, stories and archives to draw on to make your case when challenging weak institutions and encouraging colleagues and friends to join you.

Noah Haber@whaleactually.com · last yr.

If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many. Those weaknesses are points of leverage for people trying to tear the whole thing down.

If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many. Those weaknesses are points of leverage for people trying to tear the whole thing down.

When you are imagining a plucky smart investigator epidemiologist going to who knows where to investigate an outbreak like in the movies, you are thinking about the Epidemic Intelligence Service (EIS). I would strongly recommend avoiding the potato salad for a long while.

Melody Schreiber@melodyschreiber.com · last yr.

BREAKING: About 5,200 staff are being laid off across health agencies. That includes 1,250 people at CDC, including all first-year EIS officers — “disease detectives.” Employees are bracing for more layoffs in coming days.