Reproduce “Revisiting Text Ranking in Deep Research” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton
A systematic re-evaluation of text ranking methods in deep research environments demonstrates the benefits of passage-level retrieval and re-ranking for LLM agent performance. The study found that query reformulation can significantly improve neural ranker effectiveness by addressing mismatches between agent-issued queries and training data.
Key insights
- Passage-level retrieval units significantly enhance LLM agent performance, leading to higher answer accuracy and completion rates, especially for agents with limited context windows, and reduce reliance on robust document length normalization for lexical methods.
- Re-ranking consistently improves deep research agent effectiveness (recall and accuracy) across various configurations, with deeper re-ranking cut-offs yielding more pronounced benefits.
- A substantial mismatch exists between agent-issued web search-style queries and the natural-language questions neural rankers are typically trained on, which negatively impacts their performance; however, this can be effectively mitigated through context-aware query reformulation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.