Reproduce “EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haiyang Shen, Xuanzhong Chen, Wendong Xu, Yun Ma, Liang Chen, Kuan Li
EvoCode-Bench evaluates coding agents as they extend projects across changing requests, using one persistent workspace and cumulative tests for requirements that remain active. The paper’s original evaluation compares persistent multi-round MT@4 with single-attempt SR from reference-completed workspaces. Opus-4.6 scored 78.9% SR but 44.0% MT@4, ranking third on MT@4; the comparison implicates, but does not isolate, accumulated workspace state.
Key insights
- A significant gap exists between an agent's single-round instruction-following ability (SR) and its multi-turn persistent reliability (MT@4), indicating that solving isolated problems does not equate to effective iterative development.
- Agent performance consistently degrades with increasing interaction depth in multi-turn scenarios, with accumulating issues in the agent's own workspace being a primary factor rather than just the inherent difficulty of later rounds.
- Distinct failure patterns emerge across different tiers of agents: lower-tier agents struggle with basic requirement satisfaction, mid-tier agents with tracking superseded specifications, and top-tier agents with regression avoidance in advanced rounds.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.