Ian Arawjo

@ianarawjo.bsky.social

Asst Prof at Université de Montréal, Associate Member of Mila-Quebec AI Institute. PhD from Cornell InfoSci. Creator of ChainForge. Programming and culture, LLM evaluation tooling.

BREAKING: Xbox will cut 3,200 jobs as part of a major reorganization, and divest from five studios: - Compulsion and Double Fine will go indie - Ninja Theory and Undead Labs will be sold - Arkane to enter consultation process Here's the full story: www.bloomberg.com/news/article...

Microsoft’s Xbox to Cut 3,200 Jobs, Divest Five Studios in Major Overhaul

Compulsion and Double Fine studios will become independent; Undead Labs and Ninja Theory will be sold. Xbox will also look to sell or spin out Arkane Studios.

bloomberg.com

BREAKING: Several Xbox studios, including Compulsion, Ninja Theory and Double Fine, are negotiating with Xbox as they try to avoid closure. Some or all could spin off. Lots still in flux as many studios remain unsure about what's happening. Here's the latest: www.bloomberg.com/news/article...

Studios in Microsoft’s Xbox Division Brace for Closures

The studios, which include Compulsion Games and Double Fine, are in active negotiations with Xbox and may be given the chance to go independent.

bloomberg.com

We’re doing a user study to find out how adding always-on live values alongside code effects writing and debugging experiences for functional programming. See reply for sign-up details

New research thread: 1/ Error messages have always been designed, sometimes painstakingly, for humans. But now we have new "readers" for errors: agentic AI. Should this affect what PLs generate? Can we measure this experimentally? We have some preliminary results: ↵

When AI developers plot standard errors as error bars in eval charts, they're actually misleading us all—driving us to overconfident conclusions on model performance. But what should we do instead? The first investigation in the Stats for Evals blog: statsforevals.substack.com/p/why-ai-dev...

Why AI developers should use confidence intervals, not standard errors, for error bars

And a proposal for a better way to visualize AI evaluation uncertainty: gradient plots.

statsforevals.substack.com

In general, I lean against AI for qualitative research. Yet, I see ways that it could help with the tedium of QA, detect and prevent errors, and provoke questions and reflection, so am open to it being incorporated tactfully and tastefully. I just think it has been done extremely poorly so far.

The "AI for qualitative research" debate is interesting. Any HCI scholars here completely against it? Or are the HCI takes more nuanced than the open letter would suggest?

PSA for anyone submitting to HCI conferences: Do not leave the "suggested reviewers" section blank. It is your greatest (and only) way to heighten the chances your paper will be reviewed by someone who really cares about your research topic. Plus, it helps the 1AC!

I had heard people at #CHI2026 say that they didn't want AI-generated podcasts to be made on top of their work. While this perspective is totally understandable, I thought the podcasts could potentially be useful. But after looking up one of mine, I can't believe this feature was allowed to launch.

“Everyone I met knew, at some level, that AI either means that nothing matters—a kind of creeping techno-nihilism—or that everything that has always mattered—humanism, human values—is all that ever mattered, and our tool tinkering had always been a distraction.” 🙏

Amy J. Ko@amyko.phd · 4mo ago

CHI 2026 is a wrap. Here's my experience of Barcelona, CHI, and more. Thanks to the organizers for making the biggest CHI ever a reality. medium.com/bits-and-beh...

“The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding.”

Shriram Krishnamurthi@shriram.bsky.social · 4mo ago

The best piece I've read lately on AI in science is @minaskar.bsky.social's "The machines are fine. I'm worried about us." I disagree w/ his take on evaluation, but the rest is thoughtful, trenchant, and has several fine turns of phrase and useful litmus tests. ergosphere.blog/posts/the-ma...

Stats for Evals is now live, and we got a site, too: statsforevals.com We'll be posting regular investigations across the summer. For now, we're starting with the basics: comparing models and prompts. Also has resources, principles, example code, and guidance for others:

Statistics for LLM Evals

A research-backed guide to statistical methods for LLM and AI model evaluations. Learn to compare models, prompts, and agents with confidence intervals, bootstrap methods, and hypothesis tests.

statsforevals.com

It would be cool if HCI had: 1) an open reviewing platform, 2) a quid pro quo credit system like CritiqueCircle with added kudos by experts for quality reviews, 3) anonymization of reviewers (but where you can see fuzzy metrics of reviewer quality)

I've made a Substack for Stats for LLM Evals progress. We'll be releasing a website soon, but progress will be piecemeal (first release is focused on model comparison, prompt comparison, and model x prompt). Subscribe here for regular updates: substack.com/@statsforevals

Stats for Evals | Substack

Musing about statistical analysis for LLM evals. Follow for updates on the Stats for Evals project and promptstats library, including concrete investigations and recommendations.

substack.com

I don't think people understand how hard it is to work in the video-game industry right now. If you've been laid off, it can take months if not years to find new work. If you haven't been laid off, you're anxious that you will be laid off. This week's column: www.bloomberg.com/news/newslet...

It Sucks to Work in the Video-Game Industry Right Now

Even developers who successfully release big hits, like Fortnite and Battlefield, are losing their jobs

bloomberg.com

Submitting an AI-powered system paper to #UIST2026 ? Wish you could sense what reviewers were thinking, and how to maximize your chance of acceptance? Check out our #CHI2026 paper, “Reporting and Reviewing LLM-integrated Systems in HCI”, for tips and guidelines: arxiv.org/abs/2602.05128

Reporting and Reviewing LLM-Integrated Systems in HCI: Challenges and Considerations

What should HCI scholars consider when reporting and reviewing papers that involve LLM-integrated systems? We interview 18 authors of LLM-integrated system papers on their authoring and reviewing expe...

arxiv.org