2x256gb M5U: current status
Reported by r/LocalLLaMA1 source ↓
A LocalLLaMA user reports that two 256GB M5U systems work better running separate models than clustered, citing inconsistent decoding gains and memory limits. The user says GLM 3.5 Flash at Q4 handles coding better than their DeepSeek API results.
Key points
- The user runs Qwen Flash Next Q6 and GLM 3.5 Flash Q4 separately
- GLM is limited to about 260K context for three agents on 256GB, the user says
- A 32GB hot cache setting caused memory exhaustion and speeds to drop 70%
Details
The user says clustering with current tools such as TensorFold and Exo has not worked as well as running separate models. In their experience, clustering’s decoding gains are inconsistent, particularly under concurrency.
Their current setup uses Qwen Flash Next at Q6 and GLM 3.5 Flash at Q4, alongside DeepSeek through its API. The user says Qwen is faster, while GLM performs better than their DeepSeek API results for coding; these are personal assessments, not independently verified comparisons.
The user reports that 256GB is insufficient for GLM at more than 500K context, and that their setup reaches about 260K context with three agents. They also say a 32GB hot-cache setting led to memory exhaustion and a 70% speed reduction.
Written from coverage as of