Reproduce “GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yitian Zheng, Zhaorun Chen, Haoyu Li, Bo Li
GatewayBench compares flattened and hierarchical MCP gateways to assess tool exposure and performance across 524 tasks in six domains, using a pool of 11,118 tools from 251 servers. Its main 450-task DTAP evaluation ran once. With GPT-5.4, mean benign success was 87.2% versus 85.1%, and indirect attack success was 47.2% versus 30.9%; these are implementation-group averages, not guarantees.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.