Reproduce “SkillSandbox: Skill Verification via Dynamic Scenario Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Serin Kim, Kwangwook Seo, Dokyung Song, Jinyoung Yeo, Dongha Lee
SkillSandbox tests whether skills transfer advice to new tasks. A Proposer specifies each skill’s triggering condition and details to vary; a Builder creates executable scenarios; a Verifier compares the executor with and without the skill. WebShop results with synthesized evidence were 36.2% held-out success and 88.9% verdict F1, above source-task and random-task evidence. The method curates existing skills, and results are limited to tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.