🚨 [New preprint] Can AI assistants hurt the very people who depend on them? We introduce EUDAIMONIA, a benchmark grounded in a Social AI Design Code rooted in real-world harm cases. 🌐 Project page: eudaimonia-bench.github.io 📄 Paper: arxiv.org/abs/2605.30654
Frontier LLMs don't debug, they regenerate. We built PDB to measure that gap, GPT-5.1-Codex pass unit tests >76% of the time, but touch only <45% of the right lines. Even Claude Code touches only ~50%. 📄 Paper: arxiv.org/abs/2604.17338 🌐 Project: precise-debugging-benchmark.github.io
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate...
arxiv.org
At @naaclmeeting.bsky.social this week! I’ll be presenting our work on LLM domain induction with @thomason.bsky.social on Thu (5/1) at 4pm in Hall 3, Section I. Would love to connect and chat about LLM planning, reasoning, AI4Science, multimodal stuff, or anything else. Feel free to DM!
🚨 New work! LLMs often sound helpful—but fail to challenge dangerous medical misconceptions in real patient questions. We test how well LLMs handle false assumptions in oncology Q&A. 📝 Paper: arxiv.org/abs/2504.11373 🌐 Website: cancermyth.github.io 👇 [1/n]
Cancer-Myth: Evaluating AI Chatbot on Patient Questions with False Presuppositions
Cancer patients are increasingly turning to large language models (LLMs) as a new form of internet search for medical information, making it critical to assess how well these models handle complex, pe...
arxiv.org
I'll be at #NeurIPS2024! My group has papers analyzing how LLMs use Fourier Features for arithmetic and how TFs learn higher-order optimization for ICL (led by @deqing.bsky.social), plus workshop papers on backdoor detection and LLMs + PDDL (led by @billzhu.bsky.social)
✨ Excited to share our Chain-of-Questions paper #EMNLP2023: we develop a framework that trains *one T5 model* to robustly answer multistep questions by generating and answering sub-questions. Outperforms ChatGPT on DROP, HotpotQA and their contrast/adversarial sets.
