Reproduce “CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma and 1 more
Researchers from a multi-institutional collaboration defined and systematically investigated "code hallucination" in large language models (LLMs) through execution-based verification. They introduced the CodeHalu algorithm and CodeHaluEval benchmark, classifying hallucinations into four main types, and found that GPT-4 and LLaMA-3 exhibited the lowest overall hallucination rates among 17 evaluated models, with Logic Hallucinations being the most common issue.
Key insights
- Code hallucinations represent a distinct and categorizable failure mode in LLM-generated code, providing a fine-grained understanding beyond simple code errors.
- The prevalence and specific types of code hallucinations vary significantly across different LLMs, indicating diverse strengths and weaknesses in their code generation capabilities.
- Underlying causes include LLMs' token-based processing lacking higher-level structural understanding, inadequate long-distance dependency tracking, and insufficient training on resource consumption or symbolic logic.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.