Blog post on exciting research happening at MSR, including our recent work ChatBench on human-AI vs AI-alone evaluation!

Microsoft Research@msftresearch.bsky.social · last yr.

In this issue: our CHI 2025 & ICLR 2025 contributions, plus research on causal reasoning & LLMs; countering LLM jailbreak attacks; and how people use AI vs. AI-alone. Also, SVP of Microsoft Health Jim Weinstein talks rural healthcare innovation: msft.it/6013SHuu1

Looking forward to taking part in this CHI'25 panel organized by @angelhwang.bsky.social !!

Angel Hsing-Chi Hwang@angelhwang.bsky.social · last yr.

📣 Calling all #CHI2025 attendees who work with human participants: Join our panel discussion on #LLM, #simulation, #syntheticdata, and the future of human subjects research on Apr 30 (Wed), 2:10 - 3:40 PM (JP Time) Post your questions for panelists here: forms.gle/m2mXY3xFafAX...

1st post on bsky! What happens when a static benchmark comes to life? ✨ Introducing ChatBench, a large-scale user study where we *converted* MMLU questions into thousands of user-AI conversations. Then, we trained a user simulator on ChatBench to generate user-AI outcomes on unseen questions. 1/ 🧵

Check out ChatBench, our new paper+dataset. We turned AI benchmarks into user-AI chats and show that AI-alone evals often fail to predict how real humans perform with AI. @serinachang5.bsky.social @ashtonanderson.bsky.social serinachang5.github.io/assets/files... huggingface.co/datasets/mic...