Reproduce “LongEmbed: Extending Embedding Models for Long Context Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
This research introduces LONGEMBED, a new benchmark designed for long-context retrieval, and investigates efficient methods to extend the context window of existing text embedding models. The study demonstrates that strategies like NTK-Aware Interpolation can successfully extend models to handle up to 32k tokens, achieving high accuracy on complex retrieval tasks while highlighting Rotary Position Embedding's inherent advantage for extrapolating to longer sequences over Absolute Position Encoding.
Key insights
- Rotary Position Embedding (RoPE)-based architectures exhibit superior inherent extrapolation capabilities for handling longer sequences compared to Absolute Position Encoding (APE)-based models, making them more amenable to context window extension.
- Efficient fine-tuning that updates only the newly added or modified position embeddings can significantly improve the long-context retrieval performance of APE-based models, with Position Interpolation showing an advantage over Recurrent Positions.
- Specialized methods for RoPE, such as NTK-Aware Interpolation and SelfExtend, are highly effective in scaling context windows, allowing models to achieve substantial length extensions while preserving high performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.