Reproduce “Investigating Multi-layer Representations for Dense Passage Retrieval” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhongbin Xie, Thomas Lukasiewicz

Researchers from the University of Oxford and Vienna University of Technology developed Multi-layer Representations (MLR) with self-contrastive pooling, a dense passage retrieval method that effectively utilizes information from multiple encoder layers of pre-trained language models. This approach improved Top-5 accuracy on SQuAD by 3.70% and on Natural Questions by 0.67% over dual-encoder baselines, achieving these gains without increasing inference latency or memory footprint.

Key insights

  • Different layers within pre-trained language models (PLMs) encode distinct types of linguistic information, which can be effectively utilized for document representation.
  • In multi-vector settings, performance gains from using multiple layers primarily stem from a regularization effect during training, where the query's last-layer representation is implicitly encouraged to align with the most relevant document layer.
  • Pooling representations from various layers provides a more effective single-vector representation than pooling multiple representations exclusively from the last layer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.