Reproduce “Learning Retrieval Models with Sparse Autoencoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thibault Formal, Maxime Louis, Hervé Dejean, Stéphane Clinchant
SPLARE (SParse LAtent REtrieval) integrates Sparse Autoencoders (SAEs) to create sparse, latent feature representations for queries and documents, moving beyond fixed vocabulary limitations. This approach yields competitive multilingual retrieval performance on benchmarks such as MMTEB, demonstrating improved efficiency and interpretability.
Key insights
- Optimal sparse latent representations for retrieval are consistently obtained from intermediate transformer layers (around two-thirds depth), suggesting these layers capture the most relevant semantic information.
- A roughly log-linear relationship exists between SAE width (number of latent features) and retrieval effectiveness, indicating a scalable mechanism for performance improvement not available to vocabulary-based LSR.
- SPLARE demonstrates greater robustness to feature pruning at inference time and utilizes a more compact, balanced, and structured latent feature space compared to traditional SPLADE models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.