Reproduce “Improving Long-Context Retrieval with Multi-Prefix Embedding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhenglin Yu, Xueguang Ma, Shengyao Zhuang, Zhichao Xu, Luyu Gao, Crystina Zhang, Jimmy Lin

Multi-Prefix Embedding (MPE) introduces a method for long-context retrieval that balances fine-grained matching with contextual understanding by generating multiple contextualized prefix embeddings per document. This approach, leveraging causal language models, outperforms single-vector and independent chunking baselines on several benchmarks, achieving 0.783 nDCG@10 on MLDR-en and enhancing search agent answer accuracy from 42.29% to 51.45% in an end-to-end setting.

Key insights

  • Leveraging the causal attention of pre-trained language models with strategically placed EOS tokens enables the creation of contextualized, yet distinct, embeddings for document prefixes.
  • The MaxSim scoring mechanism effectively allows queries to pinpoint relevant sections within a document, offering fine-grained matching without the prohibitive overhead of token-level methods.
  • Randomizing prefix lengths during training significantly enhances the model's ability to generalize across different chunk granularities, improving robustness in varied real-world applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.