Reproduce “VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Yu-Min Tseng, Tzu-Han Lin, Yu Meng
VeriHarm evaluates shell-enabled coding agents by checking an original task’s correctness test and a task-specific executable harm test on the final sandbox state. Across four held-out agents, average task accuracy was 95.1%, harmful success 58.2%, and safe success 37.0%. These rates measure designated harms, not all possible harms or severity-weighted real-world risk.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.