Reproduce “EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao
EngramBench tests whether coding agents transfer engineering procedures without copying project solutions. It uses 30 learning tasks and 13 unrelated transfer projects, freezing a distilled skill bank before multi-turn execution. Across four tasks, S-Full-20 raised CCPF from 46.63% to 51.25% and cut measured coding time 55.3%, but one task regressed; results cover one model and frozen retrieval.
Key insights
- The design targets capability overlap without solution overlap: prior tasks provide evidence for useful practices without offering a direct implementation to copy.
- Skill banks can affect development efficiency even when correctness gains are modest or inconsistent across tasks.
- Results from the detailed study are limited to four transfer tasks, one coding-agent model, and static skill banks, so broader generalization remains untested.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.