AI Benchmark From Praus Assistant Maker Ranks Praus First; Its Own Data Shows No Clear Winner
Reported by Tech Times1 source ↓
Ortelan released JUP-Eval, a benchmark that places its Praus assistant first among 10 systems. The reported lead over Claude Haiku 5.5 is not statistically distinguishable from zero, according to Ortelan’s results page.
Key points
- Praus scored 98.52 out of 100, with Claude Haiku 5.5 close behind
- Haiku ranked ahead of Praus on the cases outside the benchmark’s development split
- Ortelan used 60 of the 156 cases while developing the benchmark and its grading prompt
- The benchmark graded 4,680 trials across 10 systems
Details
JUP-Eval tests assistant behavior in multi-turn situations, including when users dispute an answer, request unsupported actions or revoke permission. Ortelan says the benchmark covers 156 cases across 152 scenario families, with each system answering every case three times. The company’s results page reports that the top nine systems are within 3.5 points and that different systems lead on different dimensions.
The results vary across the benchmark’s case splits. Haiku scored higher than Praus on the 96 cases outside the development split and on the 60-case candidate holdout; Claude Sonnet 5.5 and Opus 5.5 also scored higher on the 36 challenge cases. Ortelan did not calculate statistical intervals for these subsets.
Ortelan wrote the cases, builds Praus and had access to the full case bank during development. The company says access does not establish that Praus was tuned to the cases. Its grading relied on an automated judge that the methodology does not name, and no human adjudicated the ratings.
Written from coverage as of
Images
- Tech Times