Reproduce “SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jennifer Williams, Dave Farris, Jeff Farris, Jiantao Jiao
SWE-Serve measures whether coding agents’ patches satisfy live model-serving requirements beyond local checks. It scores fixed repository changes with hidden tests, including end-to-end checks that launch a server and exercise public interfaces. Across 19 tasks, full scoring passed 45.9% of patches versus 69.4% without those checks, a comparison limited to SGLang on CPU or a single H100.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.