Reproduce “Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dipam Goswami, Liying Wang, Bartłomiej Twardowski, Joost van de Weijer
Researchers from the Computer Vision Center introduce Query Drift Compensation (QDC), a method enabling retrieval embedding models to learn continuously without costly re-indexing, addressing embedding drift and catastrophic forgetting. Their approach, combined with knowledge distillation, achieves an average nDCG@10 of 54.1%, matching the performance of models trained jointly on all data.
Key insights
- Catastrophic forgetting is a severe problem in Continual Document Retrieval (CDR), with naive fine-tuning leading to substantial performance degradation on older tasks.
- Re-indexing the entire document corpus, a seemingly intuitive solution, is surprisingly ineffective due to unequal embedding drift between queries and documents.
- Document embeddings, particularly for longer texts, exhibit greater drift than query embeddings when the model updates, which negatively impacts retrieval performance when re-indexed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.