Reproduce “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja and 7 more

Researchers at Scale AI developed Grade School Math 1000 (GSM1k), a new human-annotated dataset, to rigorously re-evaluate large language models' mathematical reasoning. Their study reveals that many LLMs experience up to an 8% accuracy drop on GSM1k compared to GSM8k, indicating widespread benchmark contamination and overfitting, while frontier models demonstrate greater resilience.

Key insights

  • Data contamination and overfitting are pervasive issues affecting LLM performance on existing math benchmarks, often leading to inflated scores that do not reflect true generalization.
  • Frontier LLMs, such as GPT-4, Gemini, and Claude, show significantly less overfitting, suggesting either more robust reasoning abilities that generalize well or more diligent data contamination prevention by their developers.
  • While direct data contamination contributes to overfitting, it is not the sole explanation, implying other factors like similar training data or benchmark-driven checkpoint selection also play a role.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.