Reproduce “Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiayu Wang, Weijiang Lv, Bowen Fu, Jing Fu, Jiayi Song, Lingyu Zhang, Lanxuan Xue, Luodi Chen and 3 more
A new benchmark series, Act As a Real Researcher (AARR), introduces AARRI-Bench to evaluate large language model agents on nuanced research tasks demanding professional integrity and judgment. The top-performing agent configuration, Mini-SWE-Agent with Claude-Opus-4.7, achieved a 68.3% success rate on AARRI-Bench tasks, revealing persistent struggles with subtle details like detecting fabricated data or adhering to specific ethical response formats.
Key insights
- Minimalist agent harnesses like Mini-SWE-Agent can achieve superior performance compared to more complex designs when paired with frontier LLMs, suggesting that over-engineered architectures might constrain highly intelligent models.
- LLM agents frequently achieve partial success, with a significant deficit (21.7–35.9 percentage points) between fine-grained unit test pass rates and binary 0/1 reward pass rates, indicating they often complete sub-criteria even in overall task failures.
- Qualitative analysis revealed failure modes stemming from genuine reasoning deficits (e.g., inability to detect fabricated data), harness-induced memory failures, and mismatches between agent output styles and specific grader expectations, even when the underlying reasoning was correct.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.