Reproduce “Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seonghyeon Cho, Chanjun Park

The authors propose Retrieval-Invoked Actual-Use Effect (RAE), comparing skill-enabled and skill-disabled outcomes on the same tasks where retrieval returned a skill. On DeepSeek-V3.2’s 80-task MBPP+ seed-42 sample, aggregate retrieval lift was +31.6 percentage points, while RAE was −9.1 points. RAE is conditional on retrieval and does not establish an effect across all tasks.

Key insights

  • A significant 'evaluation paradox' exists, where LLM agents can show positive aggregate performance gains from skill access but experience performance degradation on the specific tasks where skills were actually used.
  • Task-selection bias, stemming from an agent's autonomous decision to retrieve skills, distorts aggregate metrics by comparing inherently mismatched task populations.
  • High retrieval coverage does not equate to effective skill utilization; agents frequently retrieve information but fail to properly integrate or apply it, leading to 'ignored/independent' or 'misapplied' behaviors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.