Reproduce “Benchmarking Coding Agents under Evolving Requirements” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
FeatureLoop evaluates whether coding agents can add requested behavior, repair reported failures, and preserve earlier functionality. It follows repository workspaces, accumulating requirements and checking the repository baseline across three workflows: upfront requests, progressive requests, and progressive requests with correction. Across 12 models and 61 tasks, StrictPass was 64.1% with correction, while 15.4% of failed states recovered within two revisions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.