XYBench: Helping with the problem behind the question
Reported by Hugging Face and ecosistemastartup2 sources ↓
Hugging Face and research partners released XYBench, a benchmark of 8,115 questions designed to test whether AI assistants recognize mistaken assumptions and address users’ underlying goals. The lab reports that five evaluated models often focused on the request as phrased rather than the goal behind it.
Key points
- The benchmark draws on questions from Stack Overflow, Stack Exchange, curated examples and WikiHow
- On Stack Overflow, models focused on the underlying goal in 35–41% of responses
- All five evaluated models were less likely than human references to steer users away from mistaken approaches
- XYBench won the Best Resource Paper Award at EMNLP 2026
Sources
Details
XYBench was developed with researchers from Carnegie Mellon, NVIDIA and Johns Hopkins. It tests whether an assistant can identify a gap between what a user asks for and the goal they appear to be pursuing, then focus its answer on that goal. The benchmark covers programming and everyday tasks including cooking and home repair.
The 8,115 questions comprise 1,557 from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples and 6,272 generated from WikiHow guides. The researchers evaluated Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, using one response per question in a single-turn setting.
Across the four sources, all five models more often included a solution to the stated request than to the underlying goal. On Stack Overflow, they included a solution to the stated request in 75–86% of responses and to the underlying goal in 59–71%; they focused primarily on the underlying goal in 35–41%.
The researchers used Claude Sonnet 4.5 as an automated judge and compared its assessments with two human reviewers’ ratings of 500 responses. In an experiment that gave models the user’s underlying goal explicitly, Sonnet 4.6’s rate of including a solution to that goal on Stack Overflow rose from 59% to 67%, while its rate of identifying the misconception fell from 47% to 34%. The benchmark currently covers English questions and is mainly procedural.
Written from coverage as of
Images
- Hugging Face
- ecosistemastartup
- Hugging Face
- Hugging Face
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh