Reproduce “MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yizhen Jiang, Haowei Guo, Tianyi Bai, Zengjie Hu, Lu Ma, Zhengyang Zhao, Haoze Sun, Peng Pei and 1 more
MCPHallu evaluates how agents use tools across multiple servers through tasks designed to expose four failure types: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting. Across 14 models, the mean overall benchmark score was 0.642, compared with 0.458 on Unreachable Goal, where 23.2% of runs passed. These scores reveal subtype differences, not causal mechanisms or all MCP-agent failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.