Reproduce “Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tan Yu, Alexander Bukharin, Khushi Bhardwaj, Jennifer Williams, Zirui Liu, Jonathan Lingjie Li, Soumye Singhal, Joseph Jennings and 14 more
The authors introduce verifier-grounded probes to assess base coding checkpoints from successful agent trajectories, without requiring full-task agent execution. They identify the first code-changing action whose cumulative patch passes the task’s tests, then measure its likelihood, selection, or reproducibility. Across ten checkpoints, decisive-action BPB rankings correlate with post-trained SWE-bench Verified pass@1 at Spearman rho 0.964, a descriptive result without a transfer guarantee.
Key insights
- Evaluating only the action that turns a failing patch into a passing one is more predictive than averaging likelihood across every trajectory step.
- Successful trajectories provide useful context for testing coding capability without requiring a base model to independently navigate the full task and agent interface.
- These probes assess capability conditional on a successful trajectory prefix; they do not establish that a base model can independently discover that state or complete a full agentic workflow.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.