Reproduce “A Reproducibility Study of PLAID” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sean MacAvaney, Nicola Tonellotto

A reproducibility study of PLAID for efficient ColBERTv2 retrieval confirms its core effectiveness and approximation quality, while demonstrating that lexically-accelerated rerankers like LADR can offer superior efficiency-effectiveness trade-offs on densely-assessed datasets. The work also provides detailed parameter tuning guidance and a mechanistic analysis revealing PLAID's reliance on lexical matching.

Key insights

  • `ndocs` is the most influential parameter for PLAID, significantly impacting both effectiveness and efficiency, with optimal performance requiring careful, interdependent tuning of all three parameters (`nprobe`, `t_cs`, `ndocs`).
  • Lexically accelerated re-ranking techniques, especially LADR, are highly competitive, completely eclipsing PLAID's Pareto frontier on densely assessed datasets (TREC DL 2019) by offering superior efficiency-effectiveness.
  • ColBERTv2's clustered token representations exhibit strong lexical homogeneity, explaining the robust performance of lexical re-ranking baselines and suggesting a significant lexical matching component in PLAID's retrieval process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.