Reproduce “Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minghui Pan, Jiayuxuan Yang, Yuanyuan Yuan, Yu Jiang, Zhenpeng Chen
Researchers from Tsinghua University and collaborators identified that schema-formatted tool specifications in AI agents are a primary cause of Large Language Model (LLM) safety degradation, interfering with the model's inherent refusal capabilities. They developed SafeKeep, an inference-time safeguard that substantially improves agent safety and preserves task-handling by presenting flattened textual tool specifications for safety judgment.
Key insights
- Schema-formatted tool specifications, not merely their semantic content, are the primary factor degrading an LLM's safety in agent contexts by interfering with refusal-related internal representations.
- For harmful requests, schema formatting creates internal hidden-state changes that negatively align with the model's natural refusal direction, weakening its ability to reject unsafe queries.
- Causally intervening on these schema-induced activations can restore the LLM's refusal capability, confirming the mechanism by which schema formatting undermines safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.