Reproduce “STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sirui Liang, Bohan Yu, Peiyu Wang, Shiguang Guo, Wenxing Hu, Pengfei Cao, Jian Zhao, Cao Liu and 3 more
STAGE-Claw is an automated framework for state-based benchmarking of large language model (LLM)-powered agents, evaluating their performance in realistic personal computing environments. The framework automatically generates and validates diverse tasks, provides process-aware diagnostics, and showed that state-based evaluation can expose agent performance overestimation by output-only methods, with Claude-Opus-4.7 achieving the highest average score of 77.1% among 11 frontier LLMs.
Key insights
- State-based evaluation, which assesses actual persistent state changes in real applications, is crucial for accurate agent assessment, as output-only methods can significantly overestimate performance by missing real-world execution failures.
- The quality of an agent's planning, tool selection, and result verification is more critical than the sheer volume of tool invocations for successful task completion.
- Persistent memory quality, including handling noise, misleading information, and conflicts, profoundly impacts agent decisions and downstream state updates, indicating a need for advanced memory management mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.