Tessera
Folgen

Jon Mellon

@jonmellon.bsky.social

Co-director British Election Study. Political Scientist and Data Scientist. Political science methods/political behavior/causal inference. Posts do not represent employer.

919 Posts3232 Follower2629 Folgt
Chris Prosser@caprosser.com · 9h ago

Obviously, 'on average' doesn't mean polls are always-and-everywhere biased to the left - there is a lot of heterogeneity between elections - but elections are much more likely to have a leftward polling bias than a rightward one.

Bild
Chris Prosser@caprosser.com · 9h ago

Using ~8.5k election-poll-party observations from 372 elections in 32 countries, we find a persistent partisan asymmetry: polls tend to overestimate the left relative to the right. On average across elections, polls the month before an election overstated the left by about 1.4 points

Bild
Chris Prosser@caprosser.com · last wk.

No doubt you’ll all be rushing to pre-order the hardcover for the bargain price of £95 (😬), but should that prove a little steep, we’re pleased to say that the electronic version will be available open access, i.e. free!

Jon Mellon@jonmellon.bsky.social · 2mo ago

An idea I’ve been pondering is whether the academic writing community needs a pressure valve for AI: slopXiv. Institutions that presuppose human effort are in danger of getting overwhelmed but I think it’s naive to think there’s any stopping the use of LLMs

Ryan Briggs@ryancbriggs.net · 3mo ago

We have a new version of this paper out. The headline results are the same—political science must filter results heavily for statistical significance—but we've added many extensions and rewritten much of it in response to feedback (thank you!). A quick thread on updates 👇

Ryan Briggs@ryancbriggs.net · 6mo ago

I have a new paper. We look at ~all stats articles in political science post-2010 & show that 94% have abstracts that claim to reject a null. Only 2% present only null results. This is hard to explain unless the research process has a filter that only lets rejections through.

It must be very hard to publish null results Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100,000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science.
Jon Mellon@jonmellon.bsky.social · 3mo ago

This was a key point that came up in feedback. People have the intuition that nulls are a skill issue. This can be true but there’s better solutions than selecting on significance

Ryan Briggs@ryancbriggs.net · 3mo ago

3. People that did experiments also told us that they thought that null results were often indicative of a failure to manipulate. We worked out a little model with Bayesian updating to show that this point is mostly misguided. If you want evidence of dosage, you really should collect it directly.

We can now address the question that many researchers implicitly pose when faced with a null result: does a small |𝑧| constitute evidence of failed delivery? The intuition that motivates this question is partly correct. Indeed, a large |𝑧| in a plausible direction does shift posterior mass toward higher 𝑑, because Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) assigns more probability to extreme primary statistics when dose is high. However, two structural features limit how far this update goes. Consider the posterior probability that the dose was properly administered, when we have no auxiliary evidence. That probability is proportional to the distribution of the test statistic 𝑧, weighted by the prior probability of each dose level: Pr(𝑑 ∣ 𝑧) ∝ Pr(𝑧 ∣ 𝐷 = 𝑑) ⋅ Pr(𝑑). The challenge is that, since the treatment effect 𝑇 is unknown, the likelihood of 𝑧 is a mixture of two components: with and without a true effect. Pr(𝑧 ∣ 𝐷 = 𝑑) = Pr(𝑇 = 1) ⋅ Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) + Pr(𝑇 = 0) ⋅ Pr(𝑧 ∣ 𝑇 = 0). Neither component is very helpful for pinning down 𝑑. Under 𝑇 = 1, the likelihood does depend on 𝑑, but only through the same product ambiguity noted above: any combination of effect size and delivery yielding the same product is observationally equivalent. Under 𝑇 = 0, the likelihood does not depend on 𝑑 at all, so it contributes nothing to dose inference. As a result, even extreme 𝑧 statistics can lead to only modest updating about 𝑑. The intuition that “a null result implies weak delivery” is not wrong per se, but it is prior-sensitive and noisy.
Ryan Briggs@ryancbriggs.net · 3mo ago

3. People that did experiments also told us that they thought that null results were often indicative of a failure to manipulate. We worked out a little model with Bayesian updating to show that this point is mostly misguided. If you want evidence of dosage, you really should collect it directly.

We can now address the question that many researchers implicitly pose when faced with a null result: does a small |𝑧| constitute evidence of failed delivery? The intuition that motivates this question is partly correct. Indeed, a large |𝑧| in a plausible direction does shift posterior mass toward higher 𝑑, because Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) assigns more probability to extreme primary statistics when dose is high. However, two structural features limit how far this update goes. Consider the posterior probability that the dose was properly administered, when we have no auxiliary evidence. That probability is proportional to the distribution of the test statistic 𝑧, weighted by the prior probability of each dose level: Pr(𝑑 ∣ 𝑧) ∝ Pr(𝑧 ∣ 𝐷 = 𝑑) ⋅ Pr(𝑑). The challenge is that, since the treatment effect 𝑇 is unknown, the likelihood of 𝑧 is a mixture of two components: with and without a true effect. Pr(𝑧 ∣ 𝐷 = 𝑑) = Pr(𝑇 = 1) ⋅ Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) + Pr(𝑇 = 0) ⋅ Pr(𝑧 ∣ 𝑇 = 0). Neither component is very helpful for pinning down 𝑑. Under 𝑇 = 1, the likelihood does depend on 𝑑, but only through the same product ambiguity noted above: any combination of effect size and delivery yielding the same product is observationally equivalent. Under 𝑇 = 0, the likelihood does not depend on 𝑑 at all, so it contributes nothing to dose inference. As a result, even extreme 𝑧 statistics can lead to only modest updating about 𝑑. The intuition that “a null result implies weak delivery” is not wrong per se, but it is prior-sensitive and noisy.
Peter Tennant@pwgtennant.bsky.social · 4mo ago

Academics should still be forced to explain what decision would theoretically be informed their estimand! The culture of producing meaningless associations under the guise of 'interest' is extremely wasteful.

Jon Mellon@jonmellon.bsky.social · 4mo ago

Whereas in academia every chain of questions like that ultimately grounds out in “because that’s an interesting thing to know about the world” which isn’t a motivation that narrows the estimands

Jon Mellon@jonmellon.bsky.social · 4mo ago

Concrete example. Talked to someone who was wanting to estimate the ATE of a particular user action on long term engagement. I asked “why do you want to know this?” And they said “so that stakeholders know which short term metrics to try and move”

Jon Mellon@jonmellon.bsky.social · 4mo ago

An interesting difference between academic and industry data science is that there is usually a correct answer to what to the estimand *should* be in industry. This is because industry analysis needs to ultimately drive a decision whereas academic analysis has to be interesting to a community

Jon Mellon@jonmellon.bsky.social · 4mo ago

Had an agent just start writing a CSV by hand today (unclear what relation the data had to reality) when the function it wrote didn’t work correctly in its environment. Stay safe out there!