Reproduce “ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xin Gui, King Zhu, JinCheng Ren, Qianben Chen, Zekun Moore Wang, Yizhi LI, Xinpeng Liu, Xiaowan Li and 12 more

ACADREASON, a new benchmark from the OPPO AI Agent Team, evaluates advanced reasoning in large language models and agents using 50 expert-annotated problems from recent, top-tier academic papers across five diverse domains. The benchmark reveals that even state-of-the-art models like GPT-5 achieve only a 16% pass rate, exposing a substantial gap in their capabilities for complex academic research tasks.

Key insights

  • ACADREASON is exceptionally challenging for current AI models, with even the most advanced LLMs and agents showing a significant 'capability gap' in 'super-intelligent academic research tasks'.
  • Agent frameworks with external information retrieval capabilities consistently outperform standalone LLMs, demonstrating the critical role of tool augmentation for knowledge acquisition in complex academic problems.
  • Methodology-focused hints provide the most substantial performance gains, indicating that current models struggle most with deep methodological understanding and theoretical derivation, rather than just factual background or definitions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.