Reproduce “Performance Evaluation of Large Language Models in Statistical Programming” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyi Song, Kexin Xie, Lina Lee, Ruizhe Chen, Jared M. Clark, Hao He, Haoran He, Jie Min and 5 more
A study by researchers at Virginia Tech established a systematic framework for human expert evaluation of large language models in statistical SAS programming. Their evaluation of GPT-3.5, GPT-4.0, and Llama 3.1 70B across 207 tasks found that while LLMs generate syntactically plausible code, they scored significantly lower in executability (61.20%) and output correctness (51.69%) compared to code readability (94.05%).
Key insights
- LLMs demonstrated strong capabilities in generating syntactically correct and readable statistical code, yet struggled significantly with code executability and producing accurate statistical outputs.
- There was no statistically significant difference in overall performance among GPT-3.5, GPT-4.0, and Llama 3.1 70B across the evaluated statistical programming tasks.
- The human evaluation process exhibited high consistency among expert raters, validating the reliability of the assessment methodology.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.