Reproduce “LiveNewsBench: Evaluating LLM Web Search Capabilities with Freshly Curated News” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunfan Zhang, Kathleen McKeown, Smaranda Muresan

Researchers from Columbia University introduced LIVENEWSBENCH, a dynamic benchmark that generates fresh, multi-hop question-answer pairs from recent news to evaluate Multimodal Large Language Models' (MLLMs) agentic web search capabilities. The benchmark successfully distinguishes true search from memorized knowledge, showing significant performance drops when LLMs lack internet access, and supports scalable, verifiable training data generation.

Key insights

  • LIVENEWSBENCH effectively isolates genuine search capabilities, as LLMs showed significantly lower accuracy without internet access compared to other time-sensitive benchmarks, indicating strong resistance to memorization.
  • LLM performance on the benchmark varies widely, with tool-calling reliability and the strategic extraction of information from search snippets (rather than full-page visits) being key differentiators.
  • Increasing the search budget (number of queries and visits) consistently improves LLM performance, confirming the multi-hop nature of the questions and the necessity for iterative search and reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.