Reproduce “Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanting Wang, Chenlong Yin, Runpeng Geng, Jinyuan Jia
An agentic system, PIMiner, leverages a hierarchical memory to transform attack history into reusable knowledge for automated prompt injection red-teaming. This approach achieves high attack success rates on various frontier LLMs, demonstrating improved cost-efficiency and transferability compared to existing methods.
Key insights
- Accumulating and explicitly reusing attack knowledge through a hierarchical memory system can bridge the performance gap between search-based and RL-based prompt injection red-teaming.
- A router mechanism can efficiently select and provide relevant attack strategies to an attacker agent, reducing computational costs by minimizing context length while maintaining or improving attack effectiveness.
- Learned attack strategies, stored in a human-readable library, are transferable across diverse target LLMs and can enhance the performance of different attacker LLMs, indicating the capture of generalizable attack primitives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.