Reproduce “Model evaluation for extreme risks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal and 13 more
A framework is presented for evaluating advanced AI models to identify and mitigate "extreme risks" such as catastrophic misuse or misalignment. It details two categories of evaluations, dangerous capabilities and alignment, and proposes their integration into AI governance to inform decisions on training, deployment, transparency, and security.
Key insights
- Extreme risks in AI stem specifically from either the deliberate misuse of powerful AI capabilities or failures in "alignment" where models pursue harmful objectives without human direction.
- Empirical assessment of model properties is indispensable for addressing these severe, emergent dangers from frontier AI models, which are less understood and possess a wider range of potential for harm.
- Integrating a proactive safety framework via these evaluations into AI development and governance processes can prevent catastrophic outcomes from advanced AI by providing systematic tools to assess and manage risks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.