Reproduce “Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua
A diagnostic framework based on human-annotated relevance judgments characterizes the behavior and failure modes of long-horizon search agents. The analysis found that increased search effort does not directly correlate with answer accuracy, and agent failures are primarily due to retrieval gaps (51.6-64.1%) or utilization gaps, with useful evidence often obtained early in the search process.
Key insights
- Increased search effort does not strongly correlate with accuracy; instead, retrieval recall of relevant evidence is a powerful predictor of success (Pearson r = 0.99).
- Agent failures are systematically attributed to either retrieval gaps (missing necessary evidence) or utilization gaps (misusing retrieved evidence), with their prevalence varying across agents.
- Useful evidence is typically acquired early in search trajectories, followed by a significant 'wasted tail' of redundant or unproductive search efforts, and stronger agents exhibit cleaner query behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.