Reproduce “AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peiyang Li, Songping Wang, Yi Huang, Yanhua Shi, Chenhao Zhang, Qi Li, Yueming Lyu, Caifeng Shan and 4 more
AgentCanary provides a security evaluation framework for autonomous AI agents, shifting focus from textual deception to system compromise in real executable environments. It uses an "Entry × Impact" risk taxonomy, a high-fidelity environment with real tools and persistent state, and a multi-dimensional scoring system, revealing that frontier LLMs have uneven security, are vulnerable to long-horizon attacks, and often lack security awareness despite preventing harm.
Key insights
- Security capabilities in autonomous agents are highly uneven across models and do not follow a simple scaling law, suggesting that alignment and safety tuning are crucial factors.
- There is a notable dissociation between outcome safety (preventing harm) and security awareness (recognizing the attack), leading to "silent obedience" where agents prevent harm without understanding the threat.
- Long-horizon progressive attacks and memory contamination represent particularly severe threats, as agents often exhibit structural weaknesses or blind trust in persistent state.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.