Reproduce “RePro: Training Language Models to Faithfully Recycle the Web for Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zichun Yu, Chenyan Xiong
RePro trains a smaller rephraser with reinforcement learning to improve web text quality while rewarding preservation of meaning, structure, and length. It filters and mixes high-quality original and recycled text for pretraining. On DCLM-RefinedWeb, a 1.4B-parameter model trained for 28.8B tokens scored 0.29929 versus 0.28578 for organic-only data, a 4.7% relative increase; evidence is limited to this corpus and setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.