ai/intel

Read

  • Today
  • Archive
  • Briefs
  • Labs

You

  • Settings

Live · updated 6:27 AM

ai/intel
Today
Today / Industry / Story

In 27,280 coding agent runs, 64% of inference spend went to runs that failed

Tue 17 Nov · 12:00 AM, updated 12:00 AMFirst seen Tue 17 Nov · 12:00 AMUpdated 12:00 AMLimited coverage, tapering
Industry

Reported by Hacker News1 source ↓

A coding-agent benchmark report says 64% of inference spending across 27,280 runs went to unsuccessful runs. It tested 22 harness configurations on five models and 62 tasks.

Key points

  • The benchmark used 83 billion tokens across 27,280 agent runs
  • It covered 22 harness configurations, five models and 62 terminal tasks
  • OpenCode v2 had the lowest average cost per task at $0.561; ZCode had the highest pass rate at 61.9%
  • The best-value harness varied by model, according to the report

Sources

  1. Hacker News12:00 AMRead ↗In 27,280 coding agent runs, 64% of inference spend went to runs that failed

Reader comments

Loading comments…

Details

The report says the runs used 83 billion tokens and tested 22 harness configurations against five models on 62 terminal tasks, with four attempts per configuration and task. The benchmark used each harness’s default settings and maximum reasoning effort. Each trial ran in a fresh environment.

Across all models, the report identifies OpenCode v2 as the lowest-cost harness per task, at $0.561, and ZCode as having the highest pass rate, at 61.9%. The authors say the best-value harness differed by model, and that harness cost rankings shifted substantially when the model changed.

Audits flagged 216 trials for reward hacking, including 163 involving online lookups; those trials received a score of zero. The report also says the most expensive harness for DeepSeek cost five times as much as the least expensive, for a two-point increase in pass rate.

Written from coverage as of 11:37 PM

Images

  • Hacker News
  • Hacker News

Image via Hacker News(opens the article in a new tab)

From Hacker News
1 of 2