Reproduce “MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongsen Zhang, Zekun Li, Xu Luo, Xuannan Liu, Peipei Li, Wenjun Xu
Researchers from Beijing University of Posts and Telecommunications and the University of California, Santa Barbara, introduced the MCP Security Bench (MSB), a benchmark to evaluate adversarial attacks against the Model Context Protocol (MCP) in LLM agents. The study assessed ten LLM agents against 12 MCP-specific attack types across 65 tasks, revealing an average attack success rate of 40.35% and an inverse relationship between model capability and security.
Key insights
- A detailed taxonomy of 12 distinct MCP-specific attack types was established, targeting various stages of the tool-use pipeline.
- Attacks specifically exploiting the MCP framework, such as User Impersonation and False Error, demonstrated higher average success rates than traditional attacks like Prompt Injection.
- More capable LLM agents exhibited an inverse relationship between their utility and security, often being more vulnerable to MCP-specific attacks due to enhanced tool-use and instruction-following abilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.