Inconsistencies in Unsloth's LiveCodeBench Results
Reported by r/LocalLLaMA1 source ↓
A community comparison of two DeepSeek-V4.1-Flash-1017 benchmark charts found that five Unsloth scores, including the Q8 baseline, rose by about 2–3 percentage points. The post says the figures alone do not establish why they changed.
Key points
- All five Unsloth scores rose between the charts, while three GSQ-RCO scores changed by varying amounts
- The first chart corresponds to 525 outcomes; the updated methodology reports 2,100 outcomes per model
- Unsloth said the earlier chart showed one run followed by additional evaluations, with settings unchanged
- The post's probability estimate assumes the original outcomes were included and independent; it says raw data is unavailable
Details
The community post reports that Unsloth's four quantizations and Q8 baseline each scored higher in the updated LiveCodeBench v6 chart. Their scores rose from 88.19–89.33% in the first chart to 91.0–91.7% in the updated one. The post says the first-chart percentages correspond to 525 outcomes, or 175 problems with three generations each; the updated methodology uses 12 generations per problem, or 2,100 outcomes per model.
The post cites an Unsloth team comment saying the earlier chart showed one run, with additional evaluations conducted later and evaluation settings unchanged. It calculates that if the original outcomes were included in the updated results, the additional nine generations would need higher success rates than the first three. The post does not provide raw evaluation data to verify that assumption.
The author estimates a combined probability of roughly 1 in 3.9–8.0 billion for the synchronized increases under a simplified model of independent sampling, unchanged success rates and independent model fluctuations. The post cautions that the models are correlated and tested on the same tasks, so this is not an actual probability absent raw data. The three GSQ-RCO results did not all move in the same direction.
Written from coverage as of