Not another tok/s post: local quants vs Claude on 9 real bugs
Reported by GitHub and r/LocalLLaMA2 sources ↓
A LocalLLaMA user tested local models and Claude on nine private C++/CUDA bugs, asking each to diagnose the cause and write a patch. Claude Opus 5.5 scored highest; two local Qwen3.8-Flash-Next versions scored close to Claude Haiku 5.5.
Key points
- Claude Opus 5.5 scored 0.74; Sonnet 5.5 scored 0.58
- Qwen3.8-Flash-Next scored 0.50 in NVFP4 and 0.48 in 3-bit
- The nine tasks came from one private GPU-inference codebase; models could not build or run patches
- Flash-Next’s 3-bit score was 0.02 below NVFP4; the 27B model used 40% more tokens in 3-bit
- The poster reports wide confidence intervals of about ±0.2; models less than roughly 0.1 apart may be a coin flip
Sources
Details
The LocalLLaMA poster tested models on nine regressions from the private history of a C++/CUDA inference engine. Each model received a bug report based on symptoms, measurements and logs, then had to identify the cause and write a patch. The tasks were run in a sandbox without a CUDA toolchain or network access, so patches could not be built or tested.
In the poster’s scoring, Claude Opus 5.5 led with 0.74 out of 1, followed by Sonnet 5.5 at 0.58. Qwen3.8-Flash-Next scored 0.50 in NVFP4 and 0.48 in 3-bit, compared with 0.47 for Claude Haiku 5.5. The poster says scores came from an automatic check against the original fixes and a blind Claude Opus 5 judge; the measures correlated at 0.79 per patch and ranked models in the same order.
For Flash-Next, the 3-bit version used about 3% more tokens per task than NVFP4, with a score 0.02 lower. The Qwen3.8-27B 3-bit version scored 0.39 versus 0.34 in NVFP4, but used 40% more tokens per task. The poster reports that this version decoded faster but took longer per task, partly due to a prefill issue later fixed.
The poster cautions that the results cover one codebase and domain, with only nine tasks and three runs per model. Confidence intervals were about ±0.2, and the poster says differences under roughly 0.1 may be a coin flip. Claude and local models used different agent harnesses; the poster also notes that the tasks and fixes are not being published.
Written from coverage as of
Images
- GitHub
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh