PyTorch 2.15 tackles the 'one bad GPU' problem
Reported by The Register1 source ↓
PyTorch 2.15 introduces a rewritten Nvidia NCCL backend intended to improve fault diagnosis and recovery in multi-GPU training. The release also adds a flight recorder and updates for Apple hardware and Nvidia Blackwell GPUs.
Key points
- The new backend limits each process group to one communicator during initialization
- A flight recorder saves operation snapshots for inspection after failures
- The legacy backend remains available as a rollback option; PyTorch plans to remove it in 2.16
- The release includes parallel-processing improvements for Apple hardware and support for Nvidia Blackwell GPUs
Details
The rewritten NCCL backend is designed to help training jobs recover when a GPU crashes or runs slowly. The Register reports that a third-party orchestrator can exclude a lagging GPU and assign a replacement worker. Previously, failures such as timeouts or deadlocks could produce only a generic error message.
The backend also changes how GPU communication is organized: each process group gets one communicator at initialization, and programs needing parallelism must dispatch additional process groups.
PyTorch 2.15 keeps the legacy ProcessGroupNCCL backend as a rollback option. The Register reports that developers plan to remove it in PyTorch 2.16, planned for early next year, if the new backend works as expected.
Other additions include a Python transport API for asynchronous one-way data movement and parallel-processing improvements for Apple hardware. The release also adds support for Nvidia's next-generation Blackwell GPUs.
Written from coverage as of
Images
- The Register
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh