Reproduce “LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Zhao, Rongzhi Zhang, Lingchuan Liu, Hao Yang, Xunliang Cai, Xi Su
LoHoSearch introduces an automated, knowledge graph-driven benchmark designed to evaluate long-horizon reasoning and context management in search agents, pushing beyond the difficulty limitations of human-authored datasets. State-of-the-art models achieved less than 35% accuracy on this benchmark, highlighting significant challenges in complex information retrieval.
Key insights
- A knowledge graph is essential for systematically controlling search difficulty by precisely manipulating factors like entity unpopularity and relational search space size, which are difficult to achieve through manual annotation.
- Existing context management strategies demonstrate limited effectiveness on long-horizon, complex search tasks, indicating a need for more robust approaches.
- Graph-structured questions, with their cyclic dependencies and cross-constraints, are intrinsically more difficult for agents compared to tree-structured questions, as they prevent straightforward problem decomposition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.