8 agents × 250K context, concurrently, on two 3090s (~600+ tok/s total)
Reported by r/LocalLLaMA1 source ↓
A LocalLLaMA community poster reports running eight coding agents with 250K-token contexts on two RTX 3090s, with total output around 600 tokens per second. The setup keeps the full KV cache in system RAM and selects a small subset of tokens for attention at each step.
Key points
- The poster reports eight agents generating at once, each with a 250K-token context
- A six-minute test averaged 657 tokens per second, with 69–90 tokens per second per agent
- The poster reports recall of 31/32 at 250K tokens and follow-ups starting in about five seconds
- The method uses approximate attention; the poster says near-duplicate code lines can be mixed up
Details
The LocalLLaMA poster says the system runs Qwen3.8-27B W4A16 on a 3090 Ti and an RTX 3090, both power-capped at 250 watts. In a six-minute test with eight agents generating continuously at 176K–205K tokens of context, the poster measured an average of 657 tokens per second across the machine. The post also reports about 545 tokens per second for eight 250K-context agents producing explain-style output, and about 790 for copy-heavy output.
The approach keeps the full int4 KV cache in system RAM and uses a rank-64 key sketch to select roughly 1% of tokens at each decode step; the GPU holds the sketch and a smaller hot set. The poster says the patches and benchmarks are available in two GitHub repositories and require applying patches to HyperQwen. The measurements used gpu-unlock, which the poster says roughly doubles the RTX 3090’s BF16 tensor throughput.
Written from coverage as of