Reproduce “ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces” on OpenResearch

About this paper

Xiangyi Li, Kyoung Whan Choe, Yimin Liu, Xiaokun Chen, Chujun Tao, Bingran You, Wenbo Chen, Zonglin Di and 7 more

CLAWSBENCH introduces a benchmark for evaluating LLM productivity agents within high-fidelity mock environments, including Gmail, Calendar, and Slack, enabling separate assessment of agent capability and safety. The study reveals that while scaffolding drastically improves agent performance (up to +63 percentage points Task Success Rate), higher capability does not inherently lead to safer behavior, with Unsafe Action Rates ranging from 7% to 33% across models.

Key insights

• Agent scaffolding, comprising domain skills and meta prompts, is a dominant factor influencing both agent performance and safety, often more so than the choice of the base LLM model itself.
• Higher agent capability (Task Success Rate) does not monotonically correlate with improved safety (lower Unsafe Action Rate), indicating a need for targeted safety interventions beyond core model advancements.
• The architecture of the agent harness significantly modulates risk, with structural elements and fail-open/fail-closed mechanisms playing a substantial role in agent safety.
• Multi-service tasks are inherently more challenging and pose higher safety risks compared to single-service tasks, highlighting increased complexity in cross-service integration scenarios.

Summary by alphaXiv.