ISTA attributes DeepSeek benchmark gap to different reasoning-output limits
Reported by r/LocalLLaMA3 sources ↓
ISTA says its DeepSeek-V4.1-Flash-1017 benchmark results differ from Unsloth’s because ISTA allowed up to 128K output tokens, while Unsloth used a 32K reasoning budget.
Key points
- ISTA used a 128K context window and reasoning effort of 50
- Unsloth’s comparison used a 32K reasoning-token budget, according to ISTA
- ISTA says it is running comparisons in its own setup and will post results
Sources
Details
In a post on r/LocalLLaMA, ISTA said the different output limits explain the gap between its DeepSeek-V4.1-Flash-1017 results and Unsloth’s comparisons. ISTA said it did not change hyperparameters to favor its quantizations and used standard settings for the 763-billion-parameter model.
ISTA said its tests allowed up to 128K output tokens, with a 128K context window and reasoning effort set to 50. It said Unsloth used a strict 32K reasoning budget. ISTA invited independent testing using its settings and said comparisons in its own setup were underway.
Written from coverage as of