Reproduce “AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Cheng, Alfin Wijaya Rahardja, Mengshi Zhang, Zihao Chen, Zhenpeng Chen, Yiling Lou
AgentBug-Smith turns reports of bugs in an LLM agent’s harness into executable tests by constructing controlled environments, generating failure-reproducing tests, and checking that they fail on buggy code but pass on patched code. On 225 preselected issues, DeepSeek-v3.2 reproduced 63 (28.00%), versus 31 (13.78%) with SWE-Factory, at higher API cost; this compares reproduction, not repair or issue finding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.