Top 7 Decision Models Benchmark: Evaluation Across 2,826 Real Tech Decisions
Reported by r/LocalLLaMA1 source ↓
A community report evaluates seven decision models on 2,826 software-engineering decisions drawn from GitHub, Bugzilla and the CVE database. It reports a three-model tie for highest balanced accuracy, while recommending Perplexity Decider V1.2 overall based on a combination of accuracy, cost and licensing.
Key points
- Liquid d1-ultra, Nace Drex Max v1.1 and Perplexity Decider V1.2 each scored 65.6% or higher in balanced accuracy
- GPT-6 Luna Decisions scored 62.8%, the lowest reported balanced accuracy
- The benchmark covered six engineering tasks and used maintainer decisions as human ground truth
- Testing used inputs created after September 15, 2026, with data benchmarked through November 6
Sources
Details
The report says it tested seven specialized models on decisions sourced from GitHub, Firefox Bugzilla and the CVE database. The evaluated inputs covered six software-engineering tasks, with labels determined by maintainers. The report describes Perplexity Decider V1.2 as its top overall recommendation, citing a balance of accuracy, cost and open-source licensing; it names Nace Drex Max v1.1 as a close runner-up for domain-specific triage.
For overall balanced accuracy, the report gives Liquid d1-ultra 66.1%, Nace Drex Max v1.1 65.6% and Perplexity Decider V1.2 65.6%, describing them as statistically tied. It says all three significantly outperformed Liquid d1.1 at 63.7% and GPT-6 Luna Decisions at 62.8%. It also characterizes GPT-6 Luna Decisions as the most expensive and most overconfident, but provides no figures for those comparisons in the supplied text.
To limit data leakage, the benchmark used inputs created after September 15, 2026, and included data through November 6, 2026. The report says it removed labels and other potentially revealing metadata, tested three phrasings per question, and calculated 95% bootstrap confidence intervals and Holm-corrected pairwise tests. It covered English software-engineering inputs only; multimodal, legal and financial tasks were excluded.
The report notes that all seven models agreed with one another against the human label on 385 items, which it says may indicate noise in maintainer triage. Nace Drex Max v1.1 was accessed through Nace's direct endpoint, while the other six models were accessed through OpenRouter, so latency reflects different hosting configurations. Two items were excluded across all models because one request was rejected over an unparseable image and another was not returned.
Written from coverage as of