Reproduce “DeliverBench: Benchmarking AI Agents on Delivering Realistic Multi-Stage Professional Projects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yumou Liu, Mingju Chen, Zhoufutu Wen, Qianyu Yang, Zijie Wang, Sihan Chen, Yeqian Liu, Chenghao Yang and 16 more
DeliverBench evaluates whether agents can carry projects through to a coherent handoff. One agent works in a continuing workspace, using earlier deliverables as later materials and incorporating scheduled inputs and feedback. Across 11 projects, every system’s mean process score exceeded its mean delivery score; 69 of 77 outcomes did too, though the comparison does not isolate continuity as the cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.