Reproduce “LLMPerf: GPU Performance Modeling meets Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khoi N.M. Nguyen, Hoang Duy Nguyen Do, Huyen Thao Le, Thanh Tuan Dao
LLMPerf, developed by a research team at FPT Software AI Center and FPT University, introduces a method for predicting GPU kernel execution time using Large Language Models from static code analysis and execution settings. The approach achieved a Mean Absolute Percentage Error (MAPE) of 24.25% on a generated validation set and 46.1% on real-world OpenCL benchmarks.
Key insights
- Large Language Models (LLMs) can learn to infer GPU kernel execution time from static code and execution settings alone, without needing runtime information.
- A logarithmic transformation of execution time (binary logarithm) is effective for training models to predict values across a wide dynamic range.
- Creating diverse and balanced datasets, including methods for memory-analysis based input generation, improves LLM performance prediction accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.