Reproduce “HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yakup Koray Budanaz, Lorenzo Paleari, Marcin Copik, Alexandru Calotoiu, Timo Schneider, Benjamin Chen Ming Choong, Shriram Chandran, Lars Hulsbergen and 2 more
HPCAgent-Bench evaluates agents that optimize performance-critical kernels across loop-level, scientific-computing, and machine-learning tasks. It uses a separate judge to check correctness and time submissions, compares controlled setup changes, and tracks token cost. Across 40 matched comparisons, only two significantly changed speedup, while billed token cost changed by up to 3.0×; this does not establish effects beyond tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.