Reproduce “Combining Query Performance Predictors: A Reproducibility Study” on OpenResearch

About this paper

Sourav Saha, Suchana Datta, Dwaipayan Roy, Mandar Mitra, Derek Greene

Researchers from the Indian Statistical Institute and University College Dublin conducted a reproducibility study on Query Performance Predictor (QPP) combination, assessing its effectiveness across pre- and post-retrieval methods and modern datasets. The work reveals that predictor combination offers minimal to no benefit for pre-retrieval methods and diminishing returns for post-retrieval methods on larger datasets, with performance directly tied to the underlying correlations between predictors.

Key insights

• The generalized belief that combining QPP predictors consistently improves performance is an oversimplification, as pre-retrieval combinations often offer negligible or negative gains, especially on modern datasets.
• The success of QPP combination strategies is directly linked to the pairwise correlations between the individual predictors; complementary (low positive correlation) predictors lead to gains, while redundant (high positive correlation) or conflicting (negative correlation) predictors offer little benefit or degrade performance.
• Supervised neural QPP methods generally outperform traditional unsupervised methods, and their combination benefits are more pronounced on smaller datasets, diminishing on larger collections.

Summary by alphaXiv.