Effects of KV Cache Quantization on Qwen3.8 27B
Reported by r/LocalLLaMA1 source ↓
A community benchmark tested KV cache quantization across 35 weight quantizations and three inference engines. It found little measured change with 8-bit caches in llama.cpp and ExLlamaV3, while 4-bit caches increased divergence from the full-precision model.
Key points
- The tests covered contexts up to 64,000 tokens and compared outputs with an unquantized bf16 model
- In llama.cpp and ExLlamaV3, 8-bit caches changed KL divergence by less than 0.00012
- A 4-bit cache added measurable divergence, with the largest relative effect on more precise weights
- At 64k context, 8-bit caches used about 2.0–2.13 GiB, versus 4.00 GiB for 16-bit caches
- vLLM offered an 8-bit cache in the test but no 4-bit cache option
Details
The r/LocalLLaMA community benchmark compared 35 weight quantizations using llama.cpp, ExLlamaV3 and vLLM. It measured token-by-token KL divergence against an unquantized bf16 copy of Qwen3.8 27B, using prompts that included tool-use examples and varied text, with contexts reaching 64,000 tokens.
The report found that 8-bit caches in llama.cpp and ExLlamaV3 changed KL divergence by less than 0.00012 in either direction. vLLM’s FP8 cache added 0.0004–0.0006 across checkpoints. The author notes this small difference may be related to vLLM switching its attention backend when the FP8 cache is enabled.
Four-bit caches produced larger changes. In llama.cpp, the added KL divergence ranged from 0.0016 to 0.0027; for Q8_0 weights, it raised KL from 0.0007 to 0.0034. In long-document tests, the 4-bit cache penalty was about 15–30% higher at 16,000–32,000 tokens than in the first 4,000 tokens.
The report estimates that at 64,000 tokens, an 8-bit cache used 2.00–2.13 GiB, compared with 4.00 GiB for a 16-bit cache; a 4-bit cache used 1.13 GiB. vLLM did not support a 4-bit cache in the tests.
Written from coverage as of