Reproduce “Simple Features Beat Most AI Control Monitors” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shubhrangshu Debsarkar, Valmik Nahata

The authors test whether benchmark monitors identify backdoors or exploit label shortcuts in paired APPS code. A comment-stripped bag-of-words classifier, trained on the benchmark’s training split and never executing code, reaches 0.806 AUROC on the standard test split, above 20 of 24 open-weight monitors. This shows benchmark labels are predictable from surface cues, not that published monitors use them or detect sabotage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.