Reproduce “Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rabimba Karanjai, Aftab Hussain, Md Rafiqul Islam Rabin, Lei Xu, Weidong Shi, Mohammad Amin Alipour
This work explores the capabilities of large language models (LLMs) like Davinci and ChatGPT in automating unit test generation for C++ parallel programs leveraging OpenMP/MPI. It demonstrates that LLMs can produce compilable unit tests with high code coverage when provided with contextual guidance, though they currently struggle to generate tests that explicitly verify hierarchical parallelism-specific behaviors.
Key insights
- Providing contextual information and code templates significantly improves the compilation success of LLM-generated HPC unit tests, making it a more effective "code infilling" task.
- Current LLMs like Davinci and ChatGPT can achieve substantial line (up to 77.7%) and branch (up to 82.8%) code coverage for HPC applications.
- LLMs demonstrate limited understanding of complex parallel semantics, often failing to generate unit tests that explicitly verify hierarchical parallelism aspects (e.g., memory copy, reduction, atomic operations).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.