Reproduce “Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo
A statistical audit of SWE-bench shows that its broad leaderboard remains informative across the full submission pool but cannot reliably order the leading coding agents. Among the top systems, only 36 of 500 Verified instances distinguish the two highest-scoring entries, while model–scaffold choices produce score differences comparable to or larger than the entire top-thirty leaderboard spread.
Key insights
- Frontier systems largely solve the same instances: the top ten have an effective-size fraction of 0.33, and the top-two comparison has only 36 informative instances, despite 94% effective size across all Verified submissions.
- The median frontier nesting coefficient is 0.935, above a score-preserving random baseline of 0.774, indicating that weaker systems’ successes are usually contained within stronger systems’ successes.
- Scaffold effects are substantial and model-dependent: the median within-model scaffold range is 15.6 percentage points, while six of nine tested model–scaffold interactions remain significant after correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.