Reproduce “Measuring Coding Challenge Competence With APPS” on OpenResearch

About this paper

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik and 3 more

APPS introduces a comprehensive benchmark for evaluating large language models' ability to generate functional Python code from natural language specifications, utilizing a vast set of verifiable test cases. The benchmark demonstrates that current models can achieve some functional correctness on simpler tasks but struggle significantly with complex algorithmic challenges, highlighting the importance of fine-tuning on code and the inadequacy of traditional metrics like BLEU.

Key insights

• BLEU is an unreliable metric for evaluating code generation, often showing anti-correlation with actual functional correctness and increasing with problem difficulty even as model performance decreases.
• Fine-tuning on task-specific code datasets is crucial for code generation performance, as fine-tuned smaller models significantly outperform larger models like GPT-3 in a few-shot setting.
• Allowing models to generate multiple candidate solutions (Top-K) notably improves performance, suggesting they can produce several plausible options even if the first attempt is not fully correct.

Summary by alphaXiv.