Reproduce “SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks” on OpenResearch

About this paper

Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe and 70 more

SkillsBench evaluates whether reusable, file-based procedural packages called Skills help AI agents complete real-work tasks without retraining. It compares the same tasks with and without curated Skills across 87 tasks and 18 model–harness configurations. Average task-macro pass rate rises from 33.9% to 50.5%, but context and resources also change, so the results do not isolate procedural guidance as the cause.

Key insights

• Curated Skills consistently enhance agent performance across various LLM-harness configurations, but the magnitude of improvement is not uniform, indicating that Skill efficacy is empirically dependent on the agent stack.
• Concise, focused procedural guidance from 1-3 Skills is more effective than larger bundles or exhaustive documentation, suggesting an optimal balance in Skill design for maximizing benefits.
• Agent Skills can enable smaller LLMs to achieve or surpass the performance of larger models without Skills, offering a cost-effective method to augment less resource-intensive agents for procedural tasks.

Summary by alphaXiv.