Thinking Machines 4-bit replication: I trained a GPT end to end with a 4-bit forward pass on one B200
Reported by r/LocalLLaMA1 source ↓
A community researcher reports training GPT models with a 4-bit forward pass on one B200. The 840M-parameter run took about 4.3 minutes for 2,000 steps, versus 5.5 minutes in BF16, with similar validation loss.
Key points
- The 840M model’s validation loss was 1.691 with 4-bit training and 1.688 with BF16
- Per-step speedups were 18% at 219M parameters, 28% at 840M and 38% at 3.3B
- The researcher used NVFP4 for the forward pass, FP8 for backward, and high-precision master weights
- The 3.3B-parameter run lasted 1,000 steps and was described as undertrained
Details
The researcher posted the results on r/LocalLLaMA, saying the tests used one B200 rented through Modal, PyTorch 2.8 and TinyStories. The models had 219M, 840M and 3.3B parameters. The setup used NVFP4 values with one scale per 16 in the forward pass, FP8 with stochastic rounding in the backward pass, and high-precision master weights.
For the 840M model, 2,000 steps took about 4.3 minutes with the 4-bit forward pass and about 5.5 minutes in BF16. The researcher said the final validation losses were 1.691 and 1.688, respectively, and characterized the difference as noise at this scale. Each timing was repeated twice with nearly identical results.
The researcher noted that the tests did not cover multiple GPUs, MoE dispatch and combine, Four Over Six, or MXFP8 block scales in the backward pass. They also reported a rounding difference between Blackwell’s conversion instruction and the software reference, and said PyTorch’s attention backward is not bit-repeatable.
Written from coverage as of