Reproduce “SkillTester: Benchmarking Utility and Security of Agent Skills” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leye Wang, Zixing Wang, Anjie Xu
SkillTester is presented as a framework for systematically evaluating the utility and security of agent skills. It quantifies a skill's incremental value relative to a no-skill baseline and assesses its security posture through controlled probes, generating a utility score, a security score, and a three-level security status label for informed decision-making.
Key insights
- Agent skills are active components that significantly alter an agent's capabilities and trust boundaries, requiring dedicated evaluation beyond mere descriptive metadata.
- Quantifying a skill's true utility necessitates a comparative approach against a baseline to highlight incremental value rather than just raw task success.
- A structured suite of security probes is crucial to make hidden unsafe behaviors, permission boundary violations, and sensitive data handling issues observable in agent skills.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.