Reproduce “Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianing Geng, Ruiqi He, Zekun Fei, Biao Yi, Xuansheng Wu, Ruijie Wang, Zheli Liu, Xia Hu and 1 more
Researchers developed SigLeak, a black-box framework, to infer proprietary LLM agent skills from observable execution trajectories using benign queries, even against prompt defenses. SigLeak improved downstream task success rates by an average of 6.88 percentage points and achieved the highest average F1-score of 22.8% for skill similarity compared to baseline methods.
Key insights
- Proprietary agent skills systematically influence agent execution, leaving discernible "skill signatures" in observable trajectories that can be actively inferred.
- Contrastive analysis of agent behavior (with and without skill invocation) effectively isolates skill-specific procedural knowledge from general task requirements or base model behavior.
- An iterative, discrepancy-guided refinement process, driven by identifying evidence gaps in reconstructed skills, significantly enhances the quality of inferred skills.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.