Reproduce “Estimating Worst-Case Frontier Risks of Open-Weight LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch
OpenAI researchers developed a Malicious Fine-Tuning (MFT) methodology to rigorously assess the worst-case risks of `gpt-oss`, an open-weight large language model, in biology and cybersecurity domains. Their evaluations showed that even with adversarial fine-tuning, `gpt-oss`'s capabilities remain below internal "Preparedness High" risk thresholds and provide only marginal advancements over current open-weight models.
Key insights
- Adversaries can effectively train open-weight LLMs to disable safety refusals, achieving near 0% refusal rates on unsafe prompts while largely preserving general capabilities.
- Even with targeted malicious fine-tuning, current LLMs, including `gpt-oss` and OpenAI o3, still exhibit significant agentic capability gaps in complex, multi-step tasks like debugging biological protocols or executing end-to-end cyber operations.
- Simple pre-training data filtering for sensitive domains like biology may not be an effective long-term mitigation against determined adversaries attempting to re-add domain-specific data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.