AWS Launches HyperPod GPU Fault Detection; Its Own Guide Says Clean Metrics Do Not Clear Nodes
Reported by Tech Times1 source ↓
AWS released a HyperPod diagnostic summary that compares GPUs within training jobs and checks rules for ECC errors, Xid faults, network degradation, throttling and overheating. Its separate guide says silent data corruption can evade these signals.
Key points
- The summary identifies suspect GPUs, nodes or network links in HyperPod training jobs
- Peer comparisons cover utilization, memory use, temperature, power draw and SM clock frequency
- The summary supports clusters orchestrated by Amazon EKS or Slurm in all AWS Regions offering HyperPod
- AWS's guide says silent data corruption may trigger neither Xid errors nor ECC protection
- AWS reports that deterministic in-training audits added about 10% step time every 100 steps in one test
Details
AWS's Nov. 19 diagnostic summary uses CloudWatch's DetectGroupOutliers API to compare GPUs in a training job across five measures: utilization, memory use, temperature, power draw and SM clock frequency. A card that differs significantly from its peers is flagged as an outlier. Findings identify the job, cluster, accelerator and rank, and appear in the SageMaker Insights dashboard with evidence charts.
A second mechanism applies static rules for known faults, including ECC memory errors, Xid faults, Elastic Fabric Adapter network degradation, clock throttling and overheating. The summary is available for HyperPod clusters managed with Amazon EKS or Slurm in every AWS Region where the service is offered.
AWS's Nov. 11 guide says silent data corruption can produce incorrect results without triggering an Xid error or ECC protection, because it may affect arithmetic rather than memory and go undetected by the GPU. The Nov. 19 summary does not discuss this fault, so a clean diagnostic result does not establish that a node is free of silent corruption.
AWS recommends checks during training, including non-finite gradient checks, ring replay audits and comparisons of weight fingerprints across replicas. In an AWS test on a 953-million-parameter model across 32 H100 GPUs, deterministic audits added about 10% to step time when run every 100 steps and about 35% when run every 10 steps.
Written from coverage as of
Images
- Tech Times
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh