Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per second
Reported by Tom's Hardware and news.lavx.hu2 sources ↓
The builder moved 414 GB of Moonshot AI’s Kimi K3 model to a RAM drive. Tom’s Hardware reports generation speed rose 41% and prompt evaluation speed 26% compared with storing the model entirely on Optane.
Key points
- The RAM-drive setup reached 16.8 tokens per second for prompt evaluation, up from 13.3
- The 414 GB on the RAM drive is part of an 865 GB model file; the rest remains on Optane
- The RTX 3090s have 48 GB of combined VRAM, so most model data stays on the CPU side
- Cooling remains a concern, and the build retains two 12-watt fans for its Optane memory
Sources
Details
Tom’s Hardware reports that the builder’s comparison used the same two RTX 3090s in both configurations. With all 94 model shards on Optane, generation ran at 2.20 tokens per second; with 45 shards moved to a RAM drive, it ran at 3.11. Prompt evaluation increased from 13.3 to 16.8 tokens per second.
The model file is 865 GB, while the GPUs have 48 GB of combined VRAM. In this setup, the GPUs handle attention and other components used on each token, while routed expert tensors remain in system memory, on Optane or the RAM drive. The builder used Linux tmpfs for the RAM-drive portion; the article says cooling remains an issue, with the Optane fans still installed.
Written from coverage as of
Images
- Tom's Hardware