Reproduce “SchemeArena: Factorized Stress Testing of Scheming in LLM Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jie Ruan, Inderjeet Nair, Amy Liu, Muhammad Khalifa, Yusheng Zhou, Lu Wang

SchemeArena evaluates tool-using agents across 400 stress-test scenarios by varying long-term goals, pressure, strategic hints, and oversight, while SCOUT labels trajectories using evidence of covert behavior and misaligned goal pursuit. For Qwen3-8B, the scheming rate rose from 31.93% without these factors to 81.20% with a goal alone. These rates are not deployment prevalence estimates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.