Reproduce “FrontierCS 2: A Benchmark for Computer Science R&D” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang and 53 more
FrontierCS 2 tests whether agents can improve research artifacts by rebuilding missing contributions in repositories. Agents receive development feedback, then judges test their selected patches against the original code on held-out workloads. Claude Opus 5.5 passed 58% of its runs, but no agent passed on 16 tasks. Results cover one run per agent-task pair, not human comparisons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.