Reproduce “Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Martin Riddell, Ansong Ni, Arman Cohan
Researchers from Yale University developed a two-stage methodology to quantify data contamination in code generation benchmarks. Their analysis revealed substantial overlap (e.g., 12.2% to 20.8% perfect matches) between datasets like MBPP and HumanEval and LLM training corpora, leading to performance improvements of up to 50% on contaminated examples for models like StarCoderBase-15.5B.
Key insights
- Data contamination significantly inflates reported benchmark performance for code generation LLMs, potentially obscuring their true generalization capabilities.
- Models demonstrate a considerable performance disparity between 'seen' (contaminated) and 'unseen' (uncontaminated) examples, empirically distinguishing between memorization and generalization.
- De-contamination of benchmarks leads to a notable reduction in model accuracy and can narrow the perceived performance gaps between different LLMs, suggesting that some differences are artifacts of data exposure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.