Reproduce “AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zonghao Ying, Le Wang, Yisong Xiao, Jiakai Wang, Yuqing Ma, Jinyang Guo, Zhenfei Yin, Mingchuan Zhang and 2 more
A new benchmark, AGENTSAFE, systematically evaluates the safety of embodied vision-language model (VLM) agents against hazardous instructions, revealing vulnerabilities primarily in the planning stage. The research introduces SAFE-AUDIT, a thought-level safety module that improves task success rate by 2.22% on normal instructions and achieves the lowest planning (3.52%) and task success rates (0.48%) for hazardous tasks.
Key insights
- The planning stage represents the most critical vulnerability for embodied agents, as models often fail to recognize unsafe instructions and generate harmful but executable plans.
- Adversarial jailbreak methods designed for text-only LLMs can paradoxically reduce an embodied agent's ability to act harmfully by making instructions less clear or coherent, leading to plan rejection or failure.
- Proactive safety interventions at the conceptual 'thought' level, as demonstrated by SAFE-AUDIT, significantly outperform reactive, execution-layer defenses by preventing unsafe intentions from manifesting while maintaining task utility.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.