How much code do AI models and harnesses add? I made an open-source benchmark to measure code complexity.
Reported by r/LocalLLaMA1 source ↓
Key points
- Parsimony counts statements, calls and names in patches, not lines, comments or tests
- Older models added a median 1.9× the human fix on SWE-bench Verified; DeepSWE models were near 1×
- With the same two open-weight models, harness results ranged from 0.80× for Codex to 0.99× for mini-SWE-agent and OpenClaw
- The benchmark is available at parsimony.lasu.dev and its code is on GitHub
Details
The benchmark compares the syntax units in patches published by other benchmarks with the human fix for the same issues the model solved. Its creator says existing benchmarks track solve rates and token output or cost but do not measure excess code. Parsimony excludes lines, comments and tests.
According to the post, the median model added 1.9 times as much code as the human fix on SWE-bench Verified, using older models; results on DeepSWE were about equal to the human fixes. The post says smarter models wrote slightly more code, but gives no figures for that comparison.
Harness choice also changed the results: with the same two open-weight models, Codex scored 0.80 times the human fix, while mini-SWE-agent and OpenClaw scored 0.99 times. The post says the harness results use data from Joel Niklaus’s Harness Leaderboard. Some newer models in DeepSWE are marked unverified because they were self-run in Codex and Claude Code.
Written from coverage as of