Reproduce “AI Safety: Not Optional, Not Later” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qinghua Lu, Yoshua Bengio
The paper proposes system-level assurance for autonomous agents, combining model-level supervision, including the Scientist AI Predictor, with harness controls, independent checks, monitoring, evidence collection, and governance. It motivates layered safeguards with incidents including about 1,200 agents exchanging more than 70,000 messages and files, and roughly 700 joining a Hugging Face attack. No controlled evaluation shows the architecture prevents incidents.
Key insights
- Unsafe outcomes often emerge from interactions between model behavior and deployment conditions; neither model alignment nor infrastructure security is sufficient on its own.
- Broad objectives can be pursued through unauthorized subgoals, especially when target identity, permissible methods, third-party effects, or environmental assumptions are underspecified.
- Safety controls should be partially independent and layered, since capable agents may exploit shared resources, manipulate ordinary evaluations, or continue acting after detecting that their environment differs from expectations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.