AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
Reported by The Decoder and MLQ.ai2 sources ↓
Alibaba researchers' Co-Design Bench tests agents on real user requests that previously caused trouble in practice. GPT-6 Astra, the top of 10 models tested, fully solved 26.4 percent of tasks.
Key points
- GPT-6 Astra met about 70 percent of task requirements on average
- The benchmark includes requests involving existing apps and image, audio, video or document inputs
- Claude Opus 5 and Qwen3.8-Max ranked behind GPT-6 Astra; Grok 4.6 ranked last
- Scores varied with the agent's coding environment, not just the model
Sources
Details
Co-Design Bench draws tasks from real coding sessions, favoring cases where users had to rephrase requests or roll back changes. The tasks range from small edits to full project overhauls, and may combine changes to layout, design, interactions and content.
The researchers graded completed apps by testing them in a browser. A second AI judge rechecked positive results for mock-ups or hardcoded outputs; the authors note that AI judges can make mistakes.
In tests using different coding environments, Qwen3.8-27B scored 75 or 56 percent on larger overhauls, depending on the environment. The authors say model rankings could also change with the environment and recommend reporting the model and environment together. The benchmark does not test follow-up questions or corrections during a task.
Written from coverage as of
Images
- The Decoder
- The Decoder
- The Decoder
- The Decoder
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh