Reproduce “HPC-Coder: Modeling Parallel Programs using Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Nichols, Aniruddha Marathe, Harshitha Menon, Todd Gamblin, Abhinav Bhatele
HPC-Coder introduces a Large Language Model (LLM) specifically fine-tuned on a comprehensive dataset of high-performance computing (HPC) and scientific C/C++ code. The model demonstrates proficiency in generating accurate parallel code, predicting OpenMP pragmas with 97% functional correctness, and forecasting relative performance differences between code versions with up to 92% accuracy, offering a novel approach to HPC software development and performance analysis.
Key insights
- Fine-tuning general-purpose LLMs on specialized HPC code significantly improves their ability to handle domain-specific tasks, outperforming baseline models.
- Catastrophic forgetting can occur during fine-tuning, necessitating monitoring of downstream task performance in addition to perplexity to determine optimal training duration.
- LLMs can effectively correlate linguistic understanding of source code with its performance characteristics, enabling relative performance prediction without extensive empirical data collection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.