Reproduce “SkillGen: Verified Inference-Time Agent Skill Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuchen Ma, Yue Huang, Han Bao, Haomin Zhuang, Swadheen Shukla, Michel Galley, Xiangliang Zhang, Stefan Feuerriegel
SkillGen generates inspectable reusable skills from an agent’s experience without changing model weights. It contrasts failed trajectories with nearby successes, then tests candidate skills against identical verification inputs, counting repairs and regressions. Across eight models, average accuracy increased, including GPT-5.4-Nano from 54.27% to 64.35%. However, the gate uses construction-time verification data, and accepted skills can still regress on held-out cases.
Key insights
- Leveraging contrastive analysis between successful and failed agent trajectories provides more precise diagnostic signals for skill synthesis than analyzing successes or failures in isolation.
- Explicitly modeling skill synthesis as an intervention, with an empirical verification gate, is crucial for ensuring that generated skills yield a net positive effect and prevent unintended performance regressions.
- Iterative refinement guided by structured feedback from verification allows for the transformation of unstable candidate skills into robust, high-quality, and reliable final skills.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.