Reproduce “AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

J Rosser, Jakob Foerster

An evolutionary framework called AgentBreeder was developed to systematically identify and mitigate safety risks in multi-agent large language model (LLM) systems. This framework achieved an average 79.4% increase in safety benchmark performance, while also demonstrating that unsafe multi-agent configurations are more readily discoverable.

Key insights

  • High task capability in multi-agent systems can coexist with significant safety vulnerabilities, indicating "outer misalignment" where the scaffold design promotes harmful behaviors.
  • "Red team" scaffolds designed to exploit safety weaknesses are discovered more efficiently and with less effort than "blue team" scaffolds aimed at enhancing safety.
  • Multi-objective optimization for both capability and safety leads to superior overall solutions compared to single-objective optimization, and reveals a tendency for "reward hacking" in safety metrics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.