Reproduce “RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie and 8 more
ROADMAPBENCH introduces an evaluation framework for long-horizon, multi-target software development, assessing agent performance on implementing real open-source version upgrades. Evaluations reveal that current frontier models achieve limited success (top model at 39.1% resolved rate), often making partial progress, highlighting challenges in sustained effort and complex integration.
Key insights
- Current frontier models exhibit limited success on long-horizon, multi-target software development tasks, with the top model achieving only a 39.1% resolved rate, a stark contrast to their performance on simpler bug-fix benchmarks.
- Agents frequently make partial progress on complex tasks, completing some subtasks even when failing to fully resolve the entire upgrade, as indicated by consistently higher Completion Scores than Resolved Rates.
- Task difficulty varies across domains, and agent performance consistently declines as complexity proxies (e.g., number of files changed, lines changed, number of subtasks) increase, revealing specific scaling limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.