Reproduce “Agent Plasticity: Measuring Self-Improvement Through Experience” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Harman Singh, Anton Bakhtin, Rulin Shao, Gabriel Synnaeve, Ilia Kulikov, Rob Fergus, Sanjeev Arora, Kurt Keutzer and 3 more
The authors measure whether fixed-weight agents turn experience into improvements on later games. Agents revise persistent artifacts, then face held-out games; agent plasticity is held-out score gain per dollar of learning cost. Across four equally weighted in-distribution game settings, GPT-5.6 Sol has the highest estimated saturation plasticity, 298 score percentage points per $1,000, but estimates depend on the protocol and are not universal rankings.
Key insights
- Plasticity is a property of an agent under a specified learning protocol, not a universal measure of a model’s ability to learn.
- Endpoint performance and learning efficiency can diverge: Claude Fable 5 reached the highest fitted board-game endpoint, while GPT-5.6 Sol had the highest estimated plasticity to saturation.
- Artifact reuse is associated with score gains, but many failures still occur when relevant artifacts are used, pointing to limits in artifact quality, generalization, or application.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.