This job on my team at @joinprolific.bsky.social is still open (but not for much longer) job-boards.eu.greenhouse.io/prolific/job... If you are an academic or industry researcher and want to work at the coal-face of online research methods then take a look!

Sharing our open-letter to the research community “Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial” by @davmicrot.bsky.social, @soubhikbarari.bsky.social, Sunshine Hillygus, Trent Buskirk, and I. [1/6] osf.io/preprints/so...

Useful perspective from @andrewgordon.bsky.social on the bot problem in survey research: www.research-live.com/article/opin...

The right safeguards can protect public opinion polls from AI pollution | Opinion | Research live

Andrew Gordon says the threat of AI agents to survey research is overblown.

research-live.com

We have a new preprint that underscores some key claims here: even if one *can* design an agent that gets through a survey fine, it doesn't follow that such agents are undetectable or common. We find that they are far from common! Preprint link in thread👇

Association for Psychological Science@psychscience.bsky.social · 4mo ago

Psychological scientists are determined to figure out the best practices for online surveys, who—or what—is behind bad data, and how to best protect surveys from the new and emerging threat of #AI. @justinsulik.bsky.social

Job ad: I'm hiring for a Research scientist to join my team at @joinprolific.bsky.social If you've ever wondered who's working on the hard questions in online research — data quality, sampling methodology, the effect of AI on how research gets done — this is that job. [1/4]

New preprint out today (osf.io/preprints/ps...). We tested whether AI agents are actually infiltrating online surveys. Spoiler alert: they aren't Thread 🧵 [1/9]

Starting today, if an AI agent is detected in your Prolific study, we’ll give you twice the cost of that non-human participant back. You pay for human data. You expect human data. We’re backing that with our 100% Human Guarantee. Learn more: www.prolific.com/100-human-gu... #AcademicSky #Research

Fantastic example of researchers working together (and the utility of rebuttals to published work). I think we all agree that this is an area we need to invest time in, but we also need to be very careful that conclusions/interpretations are warranted from the data we collect.

Christoph Strauch@cstrauch.bsky.social · 5mo ago

We recently warned of bots in online behavioral research. @achetverikov.bsky.social showed there is no evidence for that in our @joinprolific.bsky.social data - but that doesn't mean we're safe. Agentic AI can do behavioral tasks through prompting alone. Reply & videos: osf.io/3cztr/overview

Recently, van der Stigchel and colleagues posted a provocative commentary suggesting that we should be wary of bots in online behavioral data collection (🧵by @cstrauch.bsky.social here: bsky.app/profile/cstr...). But should we? Here is my response letter osf.io/preprints/ps.... 1/5

On Thursday I'll be taking part in a roundtable “How real is the LLM threat to online research in academia?” for @joinprolific.bsky.social alongside @davmicrot.bsky.social, Michael Nicholas Stagnaro, and Raluca Rilla. Sign up here: lnkd.in/em8-MpjN

How to Bot-Proof Your Online Research

This series cuts through the noise to give researchers practical, actionable strategies for protecting their work from AI agents and bots

lnkd.in

What drives retention in online longitudinal research? We conducted an experiment (N=1,798) on @joinprolific.bsky.social, orthogonally manipulating payment rate (£6–£9/hr + bonus) and session interval (1, 2, or 4 weeks) across five sessions. The findings challenge some common assumptions 👇

Hey @rory-stewart.bsky.social @alastaircampbell2.bsky.social @therestpolitics.bsky.social I asked your favourite question to 1,936 US Adults "Who do you believe is the biggest threat to global order and security?" 33.8% of Americans rate the US as the biggest threat.... 🤯

Lots of chatter about this paper currently. Its a stark warning, but at present I see this as a stark warning of what might come, not what is happening now. As a research community we need to see it as a call-to-arms to develop new strategies, NOT a call to abandon online sampling. Reasoning below

Brendan Nyhan@brendannyhan.bsky.social · 9mo ago

From my colleague; paper here dropbox.com/scl/fi/55uok...

1 in 6 Americans can't remember the last time someone hugged them. @joinprolific.bsky.social surveyed 1,917 US adults: 24.6% were hugged today, 31.7% in the past few days. But 16.7% couldn't remember, it had been so long, the memory was gone. 🧵

We asked 1,917 Americans: "Are you OK?" Only 38% said yes. 41% said things are difficult. 21% are actively struggling. But the age breakdown is wild 🧵

AI will soon expose that most published research is mediocre, irrelevant (or simply bullsh*t). Most scientists know that, but the public doesn't... What's it going to do for trust in science when they find out - how do we plan for it? 🤔 Great article by @naomioreskes.bsky.social shorturl.at/i2Hn5

AI will soon be able to audit all published research – what will that mean for public trust in science?

An AI audit of scientific research would likely expose some fraud and widespread inconsequential work. But we need to be careful it doesn’t discredit science in general.

theconversation.com

What is one question you'd dearly love to ask a representative sample of 2,000 US adults? Going to try this again because it was super interesting last time. I'll stick the best question in our next @joinprolific.bsky.social poll that goes live at the start of November. #polling #research

Plot twist: What if AI's best future might not be the insanely profitable one 🤔 Great article arguing that instead of trillion-dollar empires, we could get free, open-source models that are just "good enough" for most people. Would that be so bad? theconversation.com/generative-a... #AI #tech

Generative AI might end up being worthless — and that could be a good thing

GenAI does some neat, helpful things, but it’s not yet the engine of a new economy — and it might not ever be.

theconversation.com

The Double Squeeze hitting Trump's base. 9 months of @joinprolific.bsky.social tracking show voters report surging costs AND growing fears of US global decline. They voted for prosperity & strength. They got strain & decline. Full analysis: www.prolific.com/resources/tr... #poll #geopolitics

Super excited to share this work with the world! We wanted to put human preference at the heart of our AI model leaderboard, and to do so in a principled and representative manner. The result: HUMAINE Check out the data here and get in touch with any questions: huggingface.co/spaces/Proli...

HUMAINE Leaderboard - a Hugging Face Space by ProlificAI

This application helps you analyze human feedback to evaluate AI models. You provide feedback data, and it gives you insights to improve your models.

huggingface.co

Prolific@joinprolific.bsky.social · 11mo ago

Introducing HUMAINE: the LLM benchmark that puts real human experience first 🎯 21,352 human evaluators. 27 models. 22 demographic groups. 5 evaluation dimensions. In partnership with @hf.co. See insights: huggingface.co/spaces/Proli... More in 🧵 #ArtificialIntelligence #MLSky #LLM #Developer

Sharing a piece I wrote for The AI Journal on who should govern AI: aijourn.com/a-fine-balan... Our recent @joinprolific.bsky.social polling shows 69.7% of people think AI investment will primarily benefit corporations not the public. They're probably right. [1/3] #AI #AIGovernance #TechPolicy

📊 Our new @joinprolific.bsky.social AI User Experience Leaderboard is live! AI systems ranked by real human experience, not just technical metrics. Using Census-based sampling + MRP for results that represent the general public. Check it out here: huggingface.co/spaces/nlpet...

UX Leaderboard - a Hugging Face Space by nlpetprolific

Leaderboard of LLMs based on detailed human feedback

huggingface.co

Prolific@joinprolific.bsky.social · last yr.

📊 Introducing Prolific’s AI user experience leaderboard—the most reliable framework for evaluating AI against human preferences. We’re thrilled to share our benchmark on @hf.co, which assesses how well language models handle real-world tasks based on user experiences 👉 www.prolific.com/leaderboard

Is #Trump right to walk back the tariffs? Our latest Prolific polling data suggests a resounding yes! In the last month the number of US adults reporting an increase in the price of everyday goods has jumped by ~10%, even for #Republican voters 😬

I'm super happy with this video from Communications of the ACM on our paper about LLM use in crowd work. Our work will be the cover paper of CACM in March. cacm.acm.org/research/pre... Cc: @veniamin.bsky.social @andrewgordon.bsky.social

Has the US public's view of foreign policy changed since #DonaldTrump became president? Our Prolific tracking polls show a general perception of a less safe world since Trump took office. This perception is strongest in #Democrats, while most #Republicans see the world as safer!

How does the US public feel about the incoming #Trump administration? We asked a representative sample of 1,938 US adults on Prolific "Are you hopeful or fearful ahead of President-elect Trump coming back to office?" 🇺🇸 Overall 46% were fearful and 41% hopeful. But stark differences in subgroups 👇

How did the debate performances of #kamalaharris and #donaldtrump impact perceptions among the public (especially in swing states)? Find out in our latest Prolific US polling round up: prolific.com/resources/sw... #polling #PresidentialElection2024 #presidentialdebate