Reproduce “The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Darshan Tank, Baran Nama
The authors evaluate reusable agent skill libraries by comparing outcomes on the same tasks with and without skills. They count newly solved tasks as gains and newly broken tasks as regressions. Across 18 conditions, 553 gains and 324 regressions mean regressions offset 59% of gains. Only three Claude Code/Sonnet SpreadsheetBench comparisons remain significant after correction, limiting that result to one benchmark and stack.
Key insights
- Skill libraries incur a "regression tax," where substantial gains are offset by regressions, underscoring that minimizing failures is as critical as maximizing new successes for robust LLM agents.
- Skill-description osmosis, where skill descriptions alone alter agent behavior even without invocation, significantly contributes to both gains and regressions, revealing a bidirectional "presence-only" influence.
- Current skill designs often overemphasize procedural guidance while under-supporting critical grounding (input interpretation) and verification (output checking) stages, which remain primary sources of persistent errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.