Reproduce “BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Islem Bouzenia, Michael Pradel
BenchMeUp constructs software-engineering benchmarks from researcher specifications. It uses shared search rules and construction plans, with agents working in parallel and fresh-container replay to check generated instances. Across four tasks, 180 of 198 instances met all seven criteria, compared with 10 of 200 from Codex; this whole-pipeline comparison is limited to four tasks, three languages, and GitHub data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.