Reproduce “ScholarSearch: Benchmarking Scholar Searching Ability of LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang

Researchers at Peking University developed ScholarSearch, the first benchmark dataset specifically designed to evaluate Large Language Models' ability to perform complex, multi-step academic information retrieval. Experiments using ScholarSearch revealed that current state-of-the-art LLMs, even when equipped with browsing capabilities, achieve modest accuracy (e.g., gpt-4o-search-preview at 18.83%), indicating significant limitations in handling deep academic research queries.

Key insights

  • Reasoning based solely on pre-trained knowledge is insufficient for answering complex academic research queries, which inherently demand real-time and dynamic information acquisition.
  • Equipping LLMs with internet browsing capabilities dramatically improves their performance in academic information retrieval, validating the crucial role of external search for these tasks.
  • Even with browsing, current state-of-the-art LLMs still achieve modest overall accuracy, indicating they are not yet fully equipped to handle the comprehensive complexity, synthesis, and critical evaluation required for deep academic research.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.