Reproduce “Investigating Data Contamination for Pre-training Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, Sanmi Koyejo
This research provides a controlled investigation into data contamination during pre-training of language models, using GPT-2 models trained from scratch on custom corpora. It reveals that ground-truth contamination, which includes prompts and answers, significantly boosts performance, while text contamination has less impact. The study also demonstrates a U-shaped performance trend with increasing data repetition and highlights the inadequacy of common n-gram-based contamination definitions.
Key insights
- Ground-truth contamination, which includes prompts and answers, generally causes more significant performance improvements than text-only contamination.
- Increasing the repetition of evaluation data in the pre-training corpus shows a U-shaped performance trend, with initial gains peaking around 10 repetitions before performance declines.
- Common n-gram-based definitions are insufficient for identifying genuinely impactful contamination, as filtering documents based on these definitions often does not reduce performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.