Reproduce “SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin, Jining Luan, Zihao Rao, Xuanqi Lan, Yang Ran and 7 more
SkillAudit introduces a skill-centered assessment framework for large language model agents, which automatically generates tailored evaluation tasks to measure a skill's utility, efficiency, and safety. Applied to 226 real-world skills, the framework found an average Pass-Rate Gain of 0.183 for skills, but also identified that 7.5% of tested skills posed significant safety risks, highlighting the independence of utility and safety.
Key insights
- Skill utility varies significantly depending on the backbone agent's strength; skills provide substantial gains (mean PRG 0.648) for weak baselines but only modest gains (mean PRG 0.034) for strong baselines.
- Skills can occasionally interfere with agent execution, leading to performance regressions, with weaker agent backbones showing higher susceptibility.
- Skill utility and safety risks are largely independent dimensions (Pearson correlation 0.111), meaning a highly useful skill can still pose significant safety concerns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.