Reproduce “CUDABench: Benchmarking LLMs for Text-to-CUDA Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma and 1 more
The CUDABench benchmark assesses Large Language Models' (LLMs) capacity for generating optimized CUDA kernels directly from natural language. This work from Shanghai Jiao Tong University introduces a comprehensive dataset with diverse computing domains, input scales, and difficulty levels, along with a hardware-independent, roofline-based performance metric, revealing that current LLMs struggle with functional correctness and fully leveraging GPU hardware capabilities.
Key insights
- Current LLMs exhibit a significant disparity between syntactic correctness (high compilation rates) and semantic accuracy (low functional correctness), with most failures stemming from logical errors rather than syntax.
- LLMs consistently demonstrate a lack of domain-specific knowledge and CUDA implementation expertise, especially in complex algorithms and when explicit guidance on parallel execution patterns is not provided.
- Generated kernels from state-of-the-art LLMs show low execution performance, often utilizing only about 40% of the GPU's theoretical computational resources, indicating a systemic failure to implement hardware-aware optimizations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.