Transformers Model Integration Contracts
Reported by Hugging Face1 source ↓
Hugging Face proposes a standard for small CPU-based model tests that check real usage paths against recorded baselines. The tests flagged a UMT5 attention regression before it was fixed.
Key points
- The UMT5 regression let the decoder attend to future tokens during training
- A small UMT5 fixture flagged the bug from its introduction until the fix
- The proposed contracts test generation, training loss, gradients and optimizer updates
- The selected T5 contract's CI check took 13.4 seconds on CPU, excluding setup
- Fixture expectations require human review before they are changed
Details
Hugging Face says Transformers enabled PyTorch scaled dot-product attention for T5 on July 30, 2026, introducing a bug that allowed UMT5's decoder to attend to future tokens during training. The bug remained in the main branch until a fix on September 28. In a replay of five months of commits, the team's small UMT5 fixture flagged it throughout that period; label log-probabilities differed from the reviewed baseline by 1.22 and training loss by about 0.20.
The proposed Model Integration Contract standard packages a reduced model fixture, a defined usage path and expected results. Tests can check more than whether a model loads and runs: the example contract also checks generated token IDs, training loss, finite gradients and whether an optimizer step changes weights. Hugging Face says fixtures preserve the checkpoint's layer types and weight file format, and use the same attention path as the real model.
The UMT5 fixture used 66 MB of weights, compared with 1,227 MB for the source checkpoint. Hugging Face says the selected T5 contract's CI runner step took 13.4 seconds on CPU, excluding setup; reverting the fix on a proof branch caused the check to fail with the same differences. The company describes transformers-mic as a proposed standard and says fixtures are intended to complement, not replace, full integration tests.
Hugging Face also reports that a Llama 4 Scout fixture reduced a 108.6-billion-parameter model to 44.7 million parameters while retaining checked layer types and its MoE routing style. In tests using synthetic injected faults, random-weight Whisper fixtures caught 18 of 24, while overfit fixtures caught all 24; both missed an assisted-decoding bug exposed by the real checkpoint. The company says it has not found a real regression caught only by the overfit fixture.
Written from coverage as of
Images
- Hugging Face
- Hugging Face
- Hugging Face
- Hugging Face