Int8-vllm updated with Nemotron 3.5 Bolt 74B
Reported by r/LocalLLaMA1 source ↓
The int8-vllm maintainer says the fork now supports Nemotron Bolt checkpoints, with changes for quantized weight handling and AMD GPU model loading.
Key points
- The fork runs the original Nvidia Bolt checkpoints, according to its maintainer
- A configurable cache keeps frequently used weights in int8 and can evict their original quantized versions
- AMD Infinity Storage support loads model data directly from storage to GPU, skipping the CPU
- The maintainer reports Bolt model loading fell from about seven minutes to two on their setup
Details
The maintainer announced the update in a r/LocalLLaMA post dated November 5, 2026. They said the optimization uses the original Nvidia Bolt checkpoints and required adding fused NVFP4 and IQ2_xxs support to int8 dequantization. The fork is available on GitHub.
The maintainer said in-kernel dequantization was slow, so the update adds a configurable cache that can keep frequently used weights in int8 while evicting their original quantized versions. They also said AMD Infinity Storage support loads models directly from storage to GPU, reducing Bolt model loading time from about seven minutes to two in their setup. These performance and stability reports come from the maintainer; the post provides no independent measurements.
Written from coverage as of
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh