Reproduce “Improving Passage Retrieval with Zero-Shot Question Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

Researchers from McGill, Mila, Meta AI, Google, and the University of Washington developed the Unsupervised Passage Re-ranker (UPR), an approach that leverages pre-trained language models to evaluate the likelihood of generating a question given a passage. This method substantially enhances passage retrieval accuracy and open-domain question answering performance without requiring any task-specific labeled data for training.

Key insights

  • The generative likelihood of a PLM producing a question from a passage (`p(q|z)`) is an effective and robust measure for passage relevance re-ranking, empirically outperforming passage generation (`p(z|q)`).
  • Instruction-tuned PLMs (e.g., T0) are particularly effective for the UPR framework, suggesting that general instruction following capabilities enhance the model's ability to interpret prompts for relevance scoring.
  • Sophisticated re-ranking performance, competitive with or surpassing supervised methods, can be achieved through clever, zero-shot application of large general-purpose PLMs, bypassing the need for task-specific labeled data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.