Reproduce “Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tingyang Chen, Cong Fu, Jiahua Wu, Haotian Wu, Hua Fan, Xiangyu Ke, Yunjun Gao, Yabo Ni and 1 more
Researchers from Zhejiang University, Shopee, Alibaba Cloud Computing, and Nanyang Technological University introduce Iceberg, a task-centric benchmark suite and "Information Loss Funnel" diagnostic model for Vector Similarity Search (VSS). This work reveals that conventional VSS evaluation metrics often misrepresent real-world application performance and provides an explainable decision tree to guide optimal VSS method selection based on data characteristics.
Key insights
- The "Information Loss Funnel" revealed that Embedding Loss and Metric Misuse are the primary drivers for task-centric accuracy, while Data Distribution Sensitivity mainly impacts retrieval efficiency.
- A significant disconnect exists where high synthetic recall often fails to predict or guarantee real-world application utility, and can sometimes even inversely correlate with downstream performance.
- The optimal choice of VSS method and similarity metric is highly context-dependent, varying significantly based on dataset characteristics and the specific downstream task, rather than a universal best performer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.