Reproduce “EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian, Zhenzhao Yuan, Junlin Yang, Dianqiao Lei, Kaiyan Zhang
EnterpriseClawBench introduces a benchmark for evaluating enterprise AI agents, leveraging tasks derived from proprietary, real-world workplace sessions. Its multidimensional evaluation framework reveals that current agent systems are largely unsaturated, with the top-performing configuration achieving a score of 0.663, and identifies challenges in areas like grounded accuracy and multimodal artifact judging.
Key insights
- Current state-of-the-art enterprise agent systems, even the best configurations, remain "unsaturated" on real-world enterprise tasks, indicating significant room for improvement in handling complex business workflows.
- Agent performance is profoundly influenced by the compatibility and interaction between the base model and its "harness" (tool-use scaffolding), where mismatches can lead to substantial performance degradation.
- Evaluating skill generalization is highly complex and depends significantly on the "creator model" used to distill the skill, with varied and often high-variance outcomes depending on the creator-consumer agent compatibility.
- LLM-based judges for multimodal outputs (e.g., visual artifacts) show substantially weaker agreement with human evaluations compared to text-based judgments, highlighting a limitation in automated evaluation for certain artifact types.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.