Reproduce “q0: Primitives for Hyper-Epoch Pretraining” on OpenResearch

About this paper

Bishwas Mandal, Shmuel Berman, Akshay Vegesna, Samip Dahal

q0 uses trajectories to produce snapshots, passes information forward through chain distillation, and fits weights on a held-out fitness set to select ensemble members. On a 1.8B-parameter model trained on 100M FineWeb tokens, q0’s 256-epoch configuration achieves validation loss 3.0034, compared with 3.0476 for an eight-member baseline; this is an epoch-budget result, not a wall-clock speedup.

Key insights

• Optimal generalization in compute-rich, data-constrained environments for LLMs involves systematically exploring and leveraging a diverse population of models, rather than solely refining a single one.
• The combination of parallel trajectories, cyclic learning rates, and cycle-boundary perturbations efficiently explores the model weight space, producing a wide range of well-converged yet diverse snapshots.
• Chain distillation allows individual model capabilities to compound across training epochs by transferring 'dark knowledge' and implicit importance signals from prior teacher models to subsequent student models.

Summary by alphaXiv.