Reproduce “HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao Zhang, Yusheng Wang, Yuhao Fei, Dongyuan Li, Zian Liang, Liuyu Xiang, Hongxun Gu, Zhaofeng He
HarnessSafe provides a benchmark and a multi-stage evaluation methodology to assess persistent-state risks in LLM agent harnesses, identifying how malicious influence propagates across carriers like memory and skills. It revealed that risk containment varies across attack types and is an emergent property of the full harness-model configuration, with individual lifecycle components proving critical for attack success.
Key insights
- A new multi-stage, trace-based evaluation scheme enables granular diagnosis of attack progression, identifying precise containment points within an agent's lifecycle beyond simple pass/fail outcomes.
- Persistent-risk containment is an emergent property of the complete harness-model configuration, indicating that neither the harness nor the LLM backend alone is sufficient to predict safety outcomes.
- The benchmark provides actionable guidance for developing more effective defensive strategies by localizing where attack chains are stopped, allowing for targeted interventions earlier in the attack lifecycle.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.