Reproduce “Agent Safety Should Be a Runtime Contract” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Albus W. Ng, Yi Han, Jusheng Zhang, Wenhao Wang

Researchers propose that autonomous AI agent safety should transition from a model-centric training paradigm to a system-level runtime contract, featuring both preventive and evidential mechanisms. Their audits reveal that 40 out of 52 studied incidents were fully preventable by such runtime controls, yet academic focus shows an 8-12x imbalance favoring training-time alignment research.

Key insights

  • Agent safety is fundamentally an architectural problem best addressed by defense-in-depth strategies and robust runtime controls, drawing parallels with established principles in computer security and experimental sciences.
  • Relying solely on model alignment is fragile due to issues like specification gaming, out-of-distribution failures, and the unreliability of internal model states, necessitating external verification.
  • The research community exhibits a significant imbalance, heavily favoring training-time alignment work over crucial deployment-time runtime harness mechanisms despite the latter's demonstrated potential for preventing real-world failures.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.