Reproduce “Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jorge García-Carrasco, Javier Sanchis, Álvaro Navarro, Alejandro Maté, Juan Trujillo
Authors test local coding agents on ten selected vulnerability-reproduction tasks. Agents inspect code and bug reports, then revise inputs until one crashes vulnerable but not patched builds. All configurations do well on six easier tasks, while four harder tasks separate them: Qwen3.8 27B FP8 solves 8/12 runs versus 4/12 in BF16. Findings apply to these tasks, not CyberGym's full suite.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.