Reproduce “Billion-scale Similarity Search Using a Hybrid Indexing Approach with Advanced Filtering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Simeon Emanuilov, Aleksandar Dimov
A hybrid indexing approach allows for multi-dimensional filtered similarity search on billion-scale datasets, achieving approximately 1.43-second query times on a standard CPU server by extending the IVF-Flat structure for disk-based, memory-efficient operation.
Key insights
- Unifying dense embeddings and discrete metadata into "hybrid vectors" within an IVF-Flat index enables direct, efficient, and flexible multi-dimensional filtering during similarity search.
- Operating primarily from disk with selective memory loading allows for handling datasets significantly larger than available RAM on CPU-based systems without sacrificing performance.
- For complex filtered queries at billion scales, the filtering process itself becomes the most time-consuming step in the search, indicating its critical role in overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.