Reproduce “Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju and 4 more
ACES evaluates skills by comparing the same task with and without a target skill under a fixed declared setup. It scores traces across six normalized metrics in an equally weighted Skill Lift. Across 947 cases spanning 58 of 64 production skills and four harnesses, mean composite lift was 0.2134 and outcome-only lift was 0.1799. These are setup-specific score differences, not universal skill values.
Key insights
- Static document scanning metrics (structural and LLM-judge scores) show negligible correlation with live agent evaluation outcomes (Spearman of -0.0181 and -0.0266 respectively), proving their insufficiency for predicting runtime skill effectiveness.
- Live-agent evaluation with ACES provides substantial gains in process-oriented metrics like skill execution (+0.3263), behavior checks (+0.2983), and skill efficiency (+0.2758), revealing crucial insights into agent behavior unobservable by static analysis.
- Paired measurement of 'Skill Lift' is essential, as stronger underlying LLM models can improve baseline agent performance, making a skill's marginal value (lift) appear to shrink even if absolute task performance remains high.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.