Reproduce “CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bangya Liu, Jianhong Tu, Yueyang Pan, Hanhong Zhao, Zhe Ye, Hongwei Li, Saastha Vasan, Katherine Lambert and 14 more
CyberArena tests agents in live contests: each attacks an opponent’s service while patching its own. Across three rounds, services change and objectives rotate. In solo play, GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3 each solved 30 of 40 challenges; their round-three capture rates in contests were 52%, 37%, and 35%. This separates the agents, but does not isolate patching as the cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.