Reproduce “On Strengths and Limitations of Single-Vector Embeddings” on OpenResearch

About this paper

Archish S, Mihir Agarwal, Ankit Garg, Neeraj Kayal, Kirankumar Shiragur

Researchers from Microsoft Research India investigate the capabilities and limitations of single-vector embeddings, demonstrating that low dimensionality is not the primary cause of poor performance on complex retrieval tasks like the LIMIT dataset. Their work highlights that domain shift, tokenization, and linguistic similarity are more critical factors, while multi-vector embeddings exhibit superior performance and robustness against catastrophic forgetting and the "drowning in documents" paradox.

Key insights

• Low-dimensional single-vector embeddings are theoretically sufficient for LIMIT-like datasets, disproving the dimensionality hypothesis as the primary cause for their observed failures.
• Domain shift, tokenization artifacts, and linguistic similarity are more influential factors contributing to the poor performance of single-vector embeddings on challenging retrieval tasks.
• Multi-vector embedding models demonstrate superior robustness to domain shifts and are significantly more resistant to catastrophic forgetting and the "drowning in documents" paradox.

Summary by alphaXiv.