Reproduce “MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zixiang Liu, Wenrui Liu, Elsie Dai, Wenhan Yu, Lei Yu, Tong Yang, Jinjun Han, Hong Gao
MCPAgentBench evaluates tool selection and coordination in local simulations using authentic tool definitions and distractors. It compares tool-call sets with gold solutions, scoring exact calls with TFS and required serial or parallel order with TEFS. Claude Sonnet 4.5 scored 71.6 TFS and 57.7 TEFS averaged over four runs. These scores distinguish call correctness from execution order, not live-service performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.