Token Pooling-Aware Training for Multi-Vector Embedding Models
Reported by Hugging Face1 source ↓
Hugging Face reports that pooling-aware training retained 97.8% of unpooled NanoBEIR nDCG@10 with four times fewer document vectors, compared with 94.6% retention for a normally trained model at the same compression level.
Key points
- Pooling-aware training improved retrieval retention from 94.6% to 97.8% at the same vector budget
- With pool_factor=4, documents retain about one quarter of their original vectors
- The example uses Sentence Transformers and evaluates on NanoBEIR
Details
Hugging Face describes training a multi-vector embedding model to preserve retrieval quality after document token vectors are pooled. In its NanoBEIR example, a normally trained model retained 94.6% of its unpooled nDCG@10 with pool_factor=4; pooling-aware training retained 97.8% at the same storage budget.
The method combines retrieval objectives on unpooled and pooled representations, with each objective contributing half of the loss. The article says a straight-through estimator lets training gradients pass from pooled vectors to their member tokens, while the clustering assignments themselves are not differentiated.
The post also describes HierarchicalTokenPooling, which uses Ward clustering to merge similar token embeddings and represents each group by its mean. Pooling can reduce storage: the article estimates that compressing from 200 to 50 vectors per document cuts quantized residual storage from 12.8 KB to 3.2 KB per document at 4-bit precision and 128 dimensions.
Written from coverage as of
Images
- Hugging Face
- Hugging Face
- Hugging Face