Qwen3 4B, 8B and 14B models released with Leech lattice quantization at about 2.7 bits per parameter
Reported by r/LocalLLaMA1 source ↓
A community developer released three Qwen3 models using a custom Leech lattice quantization format. On the developer’s L40S tests, they scored higher than similarly sized IQ2_S files on several quality measures but decoded more slowly.
Key points
- The models use 2.70–2.73 bits per parameter, including scales, tables and int4 embeddings
- On MMLU, they scored 9.1, 4.1 and 2.9 points above IQ2_S, respectively
- On the L40S, decoding was 60, 64 and 34 tokens per second slower than IQ2_S
- The format is custom; llama.cpp cannot read it
Details
The developer says the format encodes groups of 24 weights as 48-bit Leech lattice codes, with additional scales, tables and int4 embeddings bringing the models to 2.70–2.73 bits per parameter. Encoding is performed offline; a custom CUDA kernel decodes the codes during matrix-vector operations. The files are available on the Hub, and the developer says users need the custom runtime, except that the 4B model can also be used with Transformers.
In tests on one L40S, the developer compared the models with llama.cpp IQ2_S files calibrated on the same data. On MMLU, the quantized Qwen3 models scored 9.1, 4.1 and 2.9 points higher at 4B, 8B and 14B. GSM8K results were 14.3 points higher at 4B, 3 points higher at 8B and tied at 14B; WikiText-2 perplexity was also lower for all three. The developer notes that the 14B IQ2_S file was about 0.12 GB smaller.
The models decoded at 227, 145 and 90 tokens per second on the L40S, trailing IQ2_S by 60, 64 and 34 tokens per second. The developer also reports higher peak memory during an 8,000-token prompt: 5.95 GB versus 3.52 GB for the 4B comparison, attributing the difference to prefill attention handling rather than model weights. The tests cover the L40S; the developer says the files ran slower than FP16 in vLLM on an A100 and that consumer GPUs have not yet been measured.
Written from coverage as of