Serving GLM-5.3 across two Nvidia DGX stations with vLLM
Reported by Hacker News and r/LocalLLaMA2 sources ↓
A Hacker News tutorial describes running Z.ai’s GLM-5.3 NVFP4 model across two connected Nvidia DGX Stations, with vLLM serving requests through one API endpoint.
Key points
- The setup uses tensor parallelism to split model weights and computation across two GB300 GPUs
- The roughly 464 GB checkpoint exceeds the 252 GB of HBM3e on one GB300 GPU
- Station A hosts the API server; Station B runs as a worker
- The tutorial disables CPU weight offload and keeps the model weights in GPU memory
Sources
Details
The tutorial uses two DGX Stations connected over ConnectX-8 RDMA to run one model service. It configures vLLM with tensor parallelism set to two, so both GPUs participate in processing each request and Station A returns responses through an OpenAI-compatible API.
The model is a quantized version of Z.ai’s mixture-of-experts model, with 753 billion parameters and 40 billion activated per token. The tutorial says the checkpoint is about 464 GB, while one GB300 GPU has 252 GB of HBM3e; each Station stores a full checkpoint copy, and vLLM partitions the weights across the GPUs when loading them.
Written from coverage as of
Images
- Hacker News
- Hacker News
- Hacker News
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh