Reproduce “Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaonan Xu, Wenjing Wu
A controlled SkillsBench study investigates how the presentation of procedural knowledge in skill documents impacts the success rate of large language model agents. It reveals that while the availability of any skill document substantially improves agent performance, varying the level of abstraction or including a single worked example has only small and statistically uncertain effects on task success.
Key insights
- Skill availability is a major factor, with any skill document substantially improving agent task success over a no-skill baseline.
- The specific presentation granularity, such as the level of abstraction (low vs. high) or the inclusion of a single worked example, does not reliably provide a further performance boost for agents.
- Optimal skill presentation might exhibit model-specific responses, as indicated by differing effect signs for abstraction level across the two evaluated LLMs, even if effects remain uncertain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.