One token at the beginning of a prompt changed 6 out of 15 LLM responses. Does it matter?
Reported by r/LocalLLaMA1 source ↓
A community experiment found that adding a beginning-of-sequence token changed 6 of 15 Bielik-Minitron 7B responses. Only one task’s pass/fail result improved, and the author says the test does not show that adding the token generally improves quality.
Key points
- Adding <s> changed responses on 6 of 15 tasks
- The pass rate rose from 11/15 tasks without BOS to 12/15 with it
- The test used Bielik-Minitron 7B v3.0 Instruct in Ollama 0.40.14
- Ollama’s template omitted the BOS token included in the model’s GGUF template
- The author did not determine whether the template difference is a bug
Details
The experiment, posted to r/LocalLLaMA, compared Bielik-Minitron 7B v3.0 Instruct Q4_K_M responses with and without a beginning-of-sequence token, written as <s>. The author used Ollama 0.40.14 at temperature zero with seed 42 and tested 15 tasks involving Polish, JSON and Python, repeating each prompt variant three times.
The standard Ollama template and a manually prepared prompt without BOS produced identical responses on all 15 tasks. Adding BOS changed the response on six tasks, but only the Polish diacritics task changed from failing to passing; overall, the reported task pass rate went from 11 of 15 without BOS to 12 of 15 with it. The author says each variant gave the same response across its three repetitions.
The author reports that adding BOS increased each prompt by one token and says the test does not establish that the token generally improves model quality. They also say they do not know whether the difference between the GGUF template and Ollama’s template is a bug.
Written from coverage as of