Reproduce “CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyang Li, Haoyang Chen, Mattia Fazzini, Caiwen Ding
CUDABeaver introduces a benchmark for testing whether language models can repair failing CUDA programs while preserving their performance, rather than merely generating replacement kernels that pass correctness checks. Across 213 tasks and seven models, the best system reached 29.1% pass@5 under a performance-aware criterion, while protocol choices—including performance thresholds, feedback, sampling, and history—substantially changed both scores and model rankings.
Key insights
- CUDA repair should be evaluated as a distinct capability from initial code generation; a fixed broken starting point makes model comparisons more reproducible and interpretable.
- Performance requirements are essential: raising the speed threshold from 0 to 1.5 reduced model scores by 8.5–40 percentage points, exposing cases of repair by degeneration.
- Evaluation protocol is part of the measured quantity: sampling caused the largest ranking shift, with Kendall’s tau of 0.47, while feedback and retained history produced model-specific effects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.