Reproduce “ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences” on OpenResearch

About this paper

Mingda Zhang, Wenjin Liu, Tiesunlong Shen, Zikai Xiao, Zhenghong Lin, Qing Xu, Erik Cambria, Xiaoying Tang and 1 more

ScienceClaw retains verified repairs as reusable program updates while keeping foundation-model parameters fixed. It links reusable strategies, called Skills, with typed executable Operators, and keeps updates only after replay and independent validation. On ScienceClaw-Eval, out-of-distribution macro success rises from 77.72% to 91.30% over seven rounds, while EvoMaster gains a 9.78-point gain; this shows held-out, same-discipline benchmark improvement, not open-ended scientific discovery.

Key insights

• Execution traces can support reusable agent improvements, but a repair should be treated as a candidate update—not a success—until it reproduces from a reset environment and improves independent tasks.
• Combining strategic Skill changes with executable Operator updates performed better than isolated update types; ablations also found replay, IID selection, and scientific-constraint checks important to the gains.
• Transfer was stronger within related task families than across them, indicating that much of the learned improvement remained discipline-specific despite positive cross-family transfer in most tested directions.

Summary by alphaXiv.