Reproduce “ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahang Lin, Wanying Ren, Wanghan Xu, Huayu Sha, Yang Shu, Zhonghang Lu, Chengjun Pan, Yi Dong and 19 more
ARE-bench evaluates whether research agents can continue projects while preserving inherited requirements. It pairs real researcher requests with replayed workspaces and scores delivery alongside project-context checklist items. For DeepSeek-V4.1-Flash, instance success rate (ISR) falls from 69.33% under delivery-only checks to 40.00% when historical and workspace requirements are added. Across 75 tasks, this shows narrow scoring can miss contextual failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.