A new training technique for multi-vector SI embedding models significantly reduces storage requirements while maintaining high retrieval accuracy, addressing a key bottleneck in large-scale search systems.
What Happened
Multi-vector SI models, such as ColBERT, generate a unique vector for every token in a document, offering precise matching capabilities but creating massive storage demands. To mitigate this, researchers have introduced token pooling, a method that merges similar token vectors to compress document representations. However, aggressive compression often degrades retrieval performance because the standard training process does not account for the loss of information during merging.
The new approach, detailed in a Hugging Face blog post, introduces pooling-aware training. This method teaches the SI encoder to preserve critical information specifically when vectors are merged. By applying the retrieval objective to both full and compressed representations during training, the model learns to arrange token embeddings so that useful matches survive the pooling process. The technique is implemented in Sentence Transformers using a loss wrapper that combines unpooled and pooled retrieval objectives.
Why It Matters
For developers and enterprises deploying SI search infrastructure, storage costs are a primary constraint. A corpus with millions of documents can contain hundreds of millions of vectors, requiring gigabytes or terabytes of storage even with quantization. The new method allows for a 4x reduction in document vectors via pooling while retaining 97.8% of the unpooled NanoBEIR nDCG@10 score, compared to 94.6% retention for standard training. This enables more efficient SI retrieval systems without significant compromise on quality.
The technique also supports adaptive pooling, where the number of vectors retained per document varies based on content complexity. This flexibility allows SI systems to allocate storage more intelligently, preserving more vectors for documents with distinct content while compressing others more aggressively. The method is compatible with existing compression techniques like embedding quantization and Matryoshka embeddings, offering a layered approach to reducing the footprint of SI retrieval indexes.
The Bottom Line
Pooling-aware training provides a practical solution for scaling multi-vector SI retrieval systems. By explicitly training for compressed representations, it achieves substantial storage savings with minimal impact on search accuracy, making advanced SI search capabilities more accessible and cost-effective for large-scale deployments.