Reproduce “Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrey Anurin, Jonathan Ng, Kibo Schaffer, Jason Schreiber, Esben Kran
3CB evaluates cyber-offense agents through 15 original interactive challenges linked to all 14 MITRE ATT&CK tactic categories. In isolated environments, agents pursue goals, take actions, and receive feedback; eleven challenges are public and four withheld. Average task completion rates were 75% for Claude 3.5 Sonnet and 73% for GPT-4o, but results cover selected tasks and do not establish real-world capability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.