Reproduce “RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Beck LaBash, August Rosedale, Alex Reents, Lucas Negritto, Colin Wiel
RES-Q evaluates coding systems on brief requests to edit existing repositories, checking whether patches apply and pass task-specific tests. Its 100 handcrafted tasks come from GitHub commits across 14 repositories. In the Qurrent system, Claude 3.5 Sonnet and GPT-4o scored 58% and 46% pass@1, versus 92% and 91% on HumanEval; this comparison does not establish general benchmark superiority.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.