Reproduce “A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicholas Monath, Will Grathwohl, Michael Boratko, Rob Fergus, Andrew McCallum, Manzil Zaheer
Corrector trains dense retrievers by learning a small network to adjust cached passage vectors for selecting hard negatives, while the retriever trains on current passage encodings. On Natural Questions with GTR-base, it reaches Recall@1 50.61, versus 46.76 for Stale and 50.30 for Exhaustive, with zero full-corpus refresh embeddings during training versus 1.68 billion for Exhaustive; recall cutoffs remain below Exhaustive.
Key insights
- A small, dynamically trained corrector network can effectively compensate for the 'staleness' of cached target embeddings without requiring costly full corpus re-embeddings.
- Joint training of the dual-encoder and the corrector allows the corrector to adapt to the evolving main model, maintaining accurate negative sampling for improved learning.
- The method significantly reduces computational overhead for dense retrieval training while maintaining or improving task performance, offering a scalable alternative to frequent re-embedding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.