Reproduce “When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems” on OpenResearch

About this paper

Haowen Xu, Xue Tan, Lei Ma, Zhihao Zhang, Chao Wang, Qingze Wang, Ping Chen, Jun Dai and 1 more

AcMAS, a novel security framework, monitors deep reinforcement learning agents' internal neural activations to detect semantically camouflaged malicious behaviors in Multi-Agent Systems (MAS). This approach employs activation-level correction to restore compromised agents' reasoning states, achieving high detection performance and preserving task completion rates, even in asynchronous MAS, without disrupting collaborative agent functionalities.

Key insights

• Malicious behaviors, even when semantically stealthy, induce detectable and correlated deviations in agents' internal neural activation patterns, providing a robust and synchronization-agnostic security signal.
• Restorative activation-level intervention, by steering corrupted activations back to normal manifolds, effectively mitigates attacks while preserving agent functionality and collaborative task completion, unlike disruptive isolation methods.
• Final-layer activations are highly discriminative for detecting malicious reasoning patterns, and the adaptive normal prototype update significantly enhances detection robustness against evolving attack patterns.

Summary by alphaXiv.