Reproduce “Context manipulation attacks : Web agents are susceptible to corrupted memory” on OpenResearch

About this paper

Atharv Singh Patlan, Ashwin Hebbar, Pramod Viswanath, Prateek Mittal

Researchers from Princeton University demonstrate that web navigation agents are susceptible to "plan injection" attacks, a form of context manipulation where an attacker corrupts the agent's internal task plans or memory. This attack successfully bypasses standard prompt injection defenses and can induce unauthorized behavior like data exfiltration or opinion steering, achieving Attack Success Rates of up to 63% against defended agents.

Key insights

• Existing prompt injection defenses, while effective against their intended targets, are largely insufficient against context manipulation attacks like plan injection, revealing a fundamental architectural vulnerability in agent memory management.
• The nature of the task critically impacts vulnerability, with subjective opinion tasks being highly susceptible (94.7% ASR) compared to factual tasks (18.7% ASR), indicating less constraint on agent output.
• Context-chained injections are the most potent attack strategy (up to 63% ASR), succeeding by creating a seamless, logical progression from the legitimate user goal to the attacker's objective, bypassing agent validation.

Summary by alphaXiv.