Reproduce “Relative Generalization Invariance of LLM Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengzhuo Zhang, Shuche Wang, Shenggui Li, Tianyu Ruan, Jianliang He, Ivor Tsang, Tianyu Pang, Chao Du and 2 more
The authors compare validation-loss profiles across examples to study how pretraining choices affect relative difficulty, introducing Relative Generalization Invariance (RGI): two models preserve pairwise loss differences up to a constant shift. In a 124M GPT comparison on FineWeb, five non-Adam optimizers reached rCCC > 0.95 and rdiff < 0.05 after 2,000 steps with shared initialization and data, whereas changing the training stream could disrupt the pattern.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.