Reproduce “Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amir M. Ebrahimi, Gopi Krishnan Rajbahadur
The authors audit CanItEdit and EDIT-Bench as proxies for instructed code editing, comparing task distributions and tests with real-world reference sources. They find both cover a narrower range of work, while EDIT-Bench tests often do not establish that unrelated behavior is preserved. Of 39 low-coverage problems, 35 reflect deliberate test strategies; four are judged inadequate.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.