Reproduce “Gemstones: A Model Suite for Multi-Faceted Scaling Laws” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sean McLeish, John Kirchenbauer, David Yu Miller, Siddharth Singh, Abhinav Bhatele, Micah Goldblum, Ashwinee Panda, Tom Goldstein
Researchers from the University of Maryland and Columbia University introduced Gemstones, an open-source suite of over 4,000 diverse Transformer checkpoints, to investigate the robustness of scaling laws across varied architectural shapes and training hyperparameters. Their analysis reveals that scaling law prescriptions are sensitive to experimental design, with deeper models generally yielding lower loss for a given FLOP count, while wider models often achieve lower loss faster in terms of GPU hours.
Key insights
- Scaling law prescriptions are sensitive to architectural design choices and training hyperparameters, challenging assumptions of universal stability.
- Optimizing for theoretical FLOPs often favors deeper models, while optimizing for actual GPU hours (wall-clock time, with tensor parallelism) can favor wider models.
- "Overtraining" smaller models on additional tokens can be an efficient strategy, resulting in only minor increases in predicted loss despite significant reductions in model size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.