DeepSeek Ultrafast
Reported by Hacker News1 source ↓
Key points
- The engine supports 1-million-token context and multi-server KV cache sharing
- The post reports 1,700–1,770 tokens per second on 4xGB300 and 1,980–2,170 on 8xB300
- DeepSeek-V4.1-Flash-Fast is available on Baseten Model APIs
Details
The Hacker News post describes DeepSeek Ultrafast as an inference engine for DeepSeek-V4.1-Flash, with support for sm_100a and sm_103a hardware, a 1-million-token context length and multi-server KV cache sharing. It says the DeepSeek-V4.1-Flash-Fast model is available through Baseten Model APIs.
For a Tetris-code generation test, the post reports decode rates of 1,700–1,770 tokens per second on 4xGB300 and 1,980–2,170 on 8xB300. Across 30 runs, it reports average step times of 3.24–3.37 milliseconds on 4xGB300 and 2.69–2.74 milliseconds on 8xB300.
The system keeps recent conversation KV caches in host memory and uses RDMA to transfer a cache between servers when a conversation reaches a different one. Redis tracks which server holds each cache; the post says this avoids recomputing the prompt and removes the need for prefix-aware routing.
Written from coverage as of