Reproduce “Over-Searching in Search-Augmented Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, Bhuwan Dhingra

This research from Apple and Duke University identifies "over-searching" in search-augmented large language models (LLMs), where excessive search tool invocation leads to computational inefficiency and impaired abstention on unanswerable queries. Researchers quantified this trade-off using new metrics like Tokens Per Correctness (TPC) and a novel benchmark, OverSearchQA, finding that search augmentation improves answer accuracy but degrades abstention capabilities.

Key insights

  • Search augmentation creates a fundamental trade-off: it consistently improves answer accuracy on answerable questions but simultaneously degrades abstention accuracy on unanswerable ones.
  • Increased model complexity and reasoning effort amplify over-searching, leading to significantly higher computational costs (TPC) without proportional gains in overall correctness.
  • Real-world retrieval corpora often lack sufficient "negative evidence" that would help models identify and abstain from unanswerable queries, contributing to search-induced confusion.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.