Reproduce “BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos and 2 more
BrowseComp tests agents’ ability to find obscure web facts through persistent searches across scattered clues. Its 1,266 short questions have brief reference answers, judged by an AI grader for semantic equivalence. OpenAI Deep Research scored 51.5%, compared with o1 at 9.9% and GPT-4o with browsing at 1.9%; Deep Research was trained on BrowseComp-like tasks, so these results concern this benchmark, not general browsing competence.
Key insights
- Browsing capability alone is insufficient - models need both strong reasoning and strategic navigation
- Models with browsing access showed higher calibration error, suggesting overconfidence
- Performance scales with increased compute and can be improved through answer aggregation
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.