tandem.cpp: Qwen3.8-Flash-Next at 97 tok/s on a single 4090 + 64 GB RAM, 3.3× llama.cpp, works for any MoE
Reported by r/LocalLLaMA1 source ↓
A community post reports that tandem.cpp, a patch set for llama.cpp, reached 97.3 decode tok/s on Qwen3.8-Flash-Next using a consumer PC with an RTX 4090 and 64 GB of RAM.
Key points
- The post reports 51.5 tok/s on gpt-oss-120b, versus 27.8 tok/s with llama.cpp in greedy decoding
- The author says other MoE models typically run 1.4–2.1× faster when they do not fit in VRAM
- On a 64k-token prompt, the post reports prompt processing 1.4–1.8× faster
Details
The benchmark used a PC with an i5-12400F, 64 GB of DDR4 memory and an RTX 4090. For Qwen3.8-Flash-Next in IQ3_S format with MTP, tandem.cpp reached 97.3 decode tok/s in greedy mode, compared with 29.8 tok/s for llama.cpp; with Qwen’s default sampling, the results were 96.1 and 29.4 tok/s.
The post describes tandem.cpp as a patch set that keeps frequently used experts on the GPU and uses the CPU for other experts. It targets single-user decoding and is tuned for Qwen3.8, while the author reports gains on other MoE models, including gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B and DeepSeek-V4.1. The post links to build instructions, run commands and results from four machines.
Written from coverage as of