Reproduce “hacktrace: behavior-supervised detection of reward hacking during code generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hao Jiang, Xin Li, Annan Wang, Yichi Zhang, Weisi Lin

HACKTRACE monitors coding agents for shortcut behavior by scoring internal generation states and combining them with static features from final files. It trains on labels for attempted shortcuts, whether or not they succeed, rather than only successful exploits. On held-out CodeContests problems, the combined monitor reaches within-problem AUC 0.997; this is ranking performance, not classification accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.