Reproduce “The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection” on OpenResearch

About this paper

Vyzantinos Repantis, Harshvardhan Singh, Tony Joseph, Cien Zhang, Akash Vishwakarma, Svetlana Karslioglu, Michael Wyatt Thot, Ameya Gawde

This research introduces Bits-over-Random (BoR), a chance-corrected selectivity metric that quantifies the true discriminative power of retrieval systems for Large Language Models (LLMs). It reveals a "99% success paradox" where high traditional success rates can be statistically indistinguishable from random selection, demonstrating that such high retrieval depths can degrade downstream LLM performance and lead to a collapse in selectivity, especially in LLM agent tool selection.

Key insights

• High observed success rates (e.g., 99-100%) can mask performance that is indistinguishable from random selection, particularly in contexts with high relevance density or large retrieval depths (the "99% success paradox").
• Increasing retrieval depth (K) beyond a certain point can lead to a collapse in meaningful selectivity, where additional documents provide negligible gains over random chance, potentially degrading downstream LLM performance.
• The principles of selectivity collapse are highly relevant to LLM agent tool selection, where small tool catalogs and high relevance density can cause even perfect selectors to achieve near-zero selectivity.

Summary by alphaXiv.