Reproduce “Semantic Vector Encoding and Similarity Search Using Fulltext Search Engines” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jan Rygl, Jan Pomikálek, Radim Řehůřek, Michal Růžička, Vít Novotný, Petr Sojka
Researchers from RaRe Technologies and Masaryk University developed a method to enable semantic vector similarity search using existing inverted-index-based fulltext search engines like Elasticsearch. This approach involves encoding dense semantic vectors into text strings for indexing and employs a two-phase search strategy, demonstrating that it significantly outperforms native keyword-based similarity search while achieving substantial speed gains through query vector filtering.
Key insights
- Dense semantic vectors can be effectively indexed and searched by traditional inverted-index-based fulltext engines by transforming them into sparse string tokens.
- Applying high-pass filtering (trimming or selecting best features) to query vectors dramatically reduces query times with only a negligible impact on search quality for moderate filtering levels.
- A two-phase search strategy, involving candidate retrieval via the fulltext engine followed by exact similarity re-ranking, provides an effective balance between search speed and quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.