Reproduce “Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
The authors present a Common Crawl extraction pipeline for mathematical and code content. It renders HTML with Lynx to preserve layout, equations, and indentation, then uses Phi-4 to remove boilerplate and standardize equations in LaTeX. The resulting corpus contains 133.26 billion tokens. In a matched 300-billion-token training test, it reached 44.20 MATH exact-match versus 31.60 for MegaMath-Web, but the comparison does not isolate pipeline components.
Key insights
- Combining a text-based browser for structural integrity with an LLM for semantic refinement effectively overcomes the limitations of traditional heuristic-based web scrapers for scientific content.
- High-quality mathematical pretraining data not only enhances math and code specific capabilities but also contributes to broader improvements in general knowledge and reasoning skills in LLMs.
- A relatively small LLM (Phi-4, 14B parameters) can efficiently and effectively perform the complex task of boilerplate removal and math standardization, making the pipeline resource-efficient.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.