I'm presenting this paper this afternoon in Malmo! Here's a peek at the poster, which previews the article's core arguments. Come find me to talk more about how we can reclaim democratic control over AI futures :) Read full paper here: doi.org/10.1609/aies...

Shreya Chowdhary@shreyachowdhary.bsky.social · 4w ago

New #AIES paper w/ @benzevgreen.bsky.social & @tisjune.bsky.social - "Deconstructing AI Hype: A Playbook for Resistance"! Read here: shreyachowdhary.net/files/decons... We argue that AI hype is a problem not because it misdescribes AI, but because it facilitates AI entrenchment.

I think a bigger issue than what benchmarks and other evaluations measure or not measure is making claims based on them that are simply not supported by what is actually being evaluated. Knowing what one can or cannot claim seems a much more common failure to me #AIES2016

I will be presenting "Data Annotation as Measurement" at #AIES2026 in Poster Session 3 (today @ 3:15PM CEST)! ojs.aaai.org/index.php/AI...

Emma Harvey@emmharv.bsky.social · 3mo ago

✨New Work✨ forthcoming at #AIES2026: 1️⃣ "Data Annotation as Measurement" by me, @allisonkoe.bsky.social, and @kizilcec.bsky.social explores how data annotation can go wrong, and how measurement theory can help improve it. 🔗: arxiv.org/pdf/2608.07297

A quick reminder that we should not conflate *confidence* with *expertise* -- one helps you to notice whether an output is correct while the other does not #AIES2026

I'm in Europe, and next week I'll be in Malmö (for #AIES until Weds) & Copenhagen speaking at the Pioneer Center for AI on Thurs www.aicentre.dk/events/20261... Thank you @gretawarren.bsky.social & @iaugenstein.bsky.social for hosting me! If you are around and want to chat/catch up, do reach out!

P1 guest talk by Responsible AI researcher Alexandra Olteanu | Event | Pioneer Centre for Artificial Intelligence

aicentre.dk

AI alignment is at the centre of the growing debates about AI systems and their failures. The impulses of AI alignment work are however *not* new. Making sure computational systems do what we want them to do has long been a core goal of computing research.

AI alignment is at the centre of the growing debates about AI systems and their failures. The impulses of AI alignment work are however *not* new. Making sure computational systems do what we want them to do has long been a core goal of computing research.

As we have discussions about e.g., “authorship quotas” for conferences and “authorship when using LLMs” it may also be worth thinking about how each of us approaches what authorship should entail. A few thoughts about how I think about this:

On (co-)authorship

TL;DR — Authorship requires both earning it and being able to defend the work.

scholarly-things.leaflet.pub

So what do AI evaluations do or should do? Wrote a brief reflection about prototypical questions evaluations often try to answer and why it is useful to have clarity about what exactly one is trying to learn about some phenomenon of interest. rigor-in-ai.leaflet.pub/3ms4ax4sojs2j

Back-to-basics: so what do evaluations do

TL;DR — Developing useful evaluations requires clarity about what exactly one is trying to learn about a phenomenon of interest.

rigor-in-ai.leaflet.pub

Wrote another brief post motivated by a reflection on how in AI research folks might not fully appreciate the care that working with unobservable constructs requires. TL;DR -- Because unobservable constructs hold ‘surplus meaning,’ your metric is not your construct.

Back-to-basics: unobservable constructs and their ‘surplus meaning’

TL;DR — Because unobservable constructs hold ‘surplus meaning,’ your metric is not your construct.

rigor-in-ai.leaflet.pub

Wrote a brief reflection on poor conceptualizations in AI work and why they matter, something I believe is so startlingly neglected. TL;DR - Poor conceptual foundations can severely undermine the credibility and reliability of knowledge claims.

Back-to-basics: on poor conceptualizations in AI work

TL;DR — Poor conceptual foundations can severely undermine the credibility and reliability of knowledge claims. (And, no, your metric is not your construct.)

rigor-in-ai.leaflet.pub

I'm thinking about writing brief reflections on a few topics (mainly from my own attempts to gain clarity about research practices I find concerning). I'm thinking about using medium, but I see many folks use substack. Are there other platforms? What's your reasoning for using a certain platform?

I have a foolproof way to make sure that your papers contain absolutely no fake references . . . . . read the papers you cite.

Come meet me at the last poster session at ICML if you are interested in chatting about LLM evaluation and pairwise comparisons! I will be presenting joint work with my advisor (Moritz Hardt). Thu Jul 9, 5:00 PM – 6:45 PM KST, Hall A (#4411) arxiv.org/abs/2606.09409

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge bi...

arxiv.org

So often folks conflate outputs with outcomes; in most settings, the system outputs are different than the outcomes of interest.

Sometimes it feels like some folks are losing the plot about what the goal of publishing research actually is. The goal is certainly not meant to be the productions of papers, but rather the production and communication of science.

Yesterday was my last day at MSR. We recently learned that our roles were eliminated, and with them our little FATE Montreal team. I joined MSR a bit over 7.5 years ago while on active chemotherapy, and being at MSR has overlapped with so much change in my life.

The deadline for the 2026 FAccT DC is next Tuesday, February 24! If you are a student working on topics relevant to the FAccT's scope, this is an opportunity to interact with a diverse set of peers and mentors! #facct2026 #facct26 #facct Details here: facctconference.org/2026/callfor...

Alexandra Olteanu@aolteanu.bsky.social · 10mo ago

We are hoping to see applications from and bring together a diverse group of students across multiple disciplinary areas! If you are a graduate student and interested in FAccT’s scope, the Doctoral Colloquium is for you! #facct2026 #facct26

The deadline for the 2026 FAccT DC is next Tuesday, February 24! If you are a student working on topics relevant to the FAccT's scope, this is an opportunity to interact with a diverse set of peers and mentors! #facct2026 #facct26 #facct Details here: facctconference.org/2026/callfor...

Alexandra Olteanu@aolteanu.bsky.social · 10mo ago

We are hoping to see applications from and bring together a diverse group of students across multiple disciplinary areas! If you are a graduate student and interested in FAccT’s scope, the Doctoral Colloquium is for you! #facct2026 #facct26

We are hoping to see applications from and bring together a diverse group of students across multiple disciplinary areas! If you are a graduate student and interested in FAccT’s scope, the Doctoral Colloquium is for you! #facct2026 #facct26

ACM FAccT@facct.bsky.social · 10mo ago

Calling all graduate students! The #FAccT2026 call is out for our Doctoral Colloquium. Join us for connections, support, mentorship, etc. Apply by February 24! facctconference.org/2026/callfor...

📣 It's again that time of the year 🤩 - our internship call for the FATE team @ MSR Montreal and our collaborators is now up! 🎉🎉 We are broadly looking for candidates interested in perceptions, evaluation, uses, and impacts of AI! #FAccT #FAccT2025 #CHI2025 #NeurIPS2025 #ACL2025

📣 It's again that time of the year 🤩 - our internship call for the FATE team @ MSR Montreal and our collaborators is now up! 🎉🎉 We are broadly looking for candidates interested in perceptions, evaluation, uses, and impacts of AI! #FAccT #FAccT2025 #CHI2025 #NeurIPS2025 #ACL2025

@zey.bsky.social's paper and talk at ICWSM'14 (my first ICWSM!) shaped my thinking on issues around data/technology and the type of problems I still spend my time working on/thinking about. Can't think of a better keynote speaker for my first NeurIPS.

On my way to @neuripsconf.bsky.social -- this is my first time at #NeurIPS and looking forward to seeing folks and presenting our paper on Rigor in AI. Do find me if you want to chat about rigor in AI, anthropomorphic AI, or evaluations #NeurIPS2025

It's the season for PhD apps!! 🥧 🦃 ☃️ ❄️ Apply to Wisconsin CS to research - Societal impact of AI - NLP ←→ CSS and cultural analytics - Computational sociolinguistics - Human-AI interaction - Culturally competent and inclusive NLP with me! lucy3.github.io/prospective-...

Unexpected (amount of) snow day

Our forthcoming NeurIPS position paper, led by @aolteanu.bsky.social, makes this argument (along with several related ones) in more depth. Rigorous AI/ML work should flow from explicit and rigorous premises, not just have a final evaluation that checks some rigor boxes. arxiv.org/abs/2506.14652

Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor

In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue th...

arxiv.org

I wish folks would use more precise terminology than "AI sycophancy." Not all validating behaviours/interactions are sycophantic. By definition, for them to be sycophantic there needs to be an underlying intention to e.g., gain advantage or favour. Intention is something AI systems do not have.

Love this analogy

Olivia Guest · Ολίβια Γκεστ@olivia.science · last yr.

People who coax chatbots into sensible answers are basically opening and closing the fridge until it contains something you wanna eat, yes, eventually you get hungrier & eat the stuff in there. But what changed was your cognition. The fridge stayed the same. You changed your mind about the contents.

This was accepted to #NeurIPS 🎉🎊 TL;DR Impoverished notions of rigor can have a formative impact on AI work. We argue for a broader conception of what rigorous work should entail & go beyond methodological issues to include epistemic, normative, conceptual, reporting & interpretative considerations

Alexandra Olteanu@aolteanu.bsky.social · last yr.

We have to talk about rigor in AI work and what it should entail. The reality is that impoverished notions of rigor do not only lead to some one-off undesirable outcomes but can have a deeply formative impact on the scientific integrity and quality of both AI research and practice 1/