Reproduce “Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing” on OpenResearch

About this paper

Wuyuao Mai, Geng Hong, Qi Liu, Jinsong Chen, Jiarun Dai, Xudong Pan, Yuan Zhang, Min Yang

Researchers at Fudan University introduced TermiBench, the first real-world benchmark for automated penetration testing, and TermiAgent, a multi-agent framework. TermiAgent effectively compromises over 50% of TermiBench's targets, achieving a 92.57% reduction in financial cost and an 81.33% reduction in execution time compared to existing methods, while also highlighting vulnerabilities in LLM safety mechanisms.

Key insights

• TermiAgent's Located Memory Activation (LMA) is crucial for performance, especially in complex, multi-service environments, reducing context load and enabling efficient, targeted decision-making.
• The Unified Exploit Descriptor (UED) approach successfully standardizes and automates the integration of diverse, 'in-the-wild' exploits, providing a robust and up-to-date exploit arsenal that significantly expands beyond traditional tools like Metasploit.
• Sophisticated agent designs, such as TermiAgent's context filtering and task decomposition, can bypass current LLM safety alignments without explicit jailbreaking, revealing a critical vulnerability in existing LLM safeguards.

Summary by alphaXiv.