Reproduce “Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marianna Nezhurina, Tomer Porian, Giovanni Pucceti, Tommie Kerssies, Romain Beaumont, Mehdi Cherti, Jenia Jitsev
Researchers from LAION and Juelich Supercomputing Center demonstrate that scaling law derivation can serve as a robust methodology for systematically comparing open foundation language-vision models and datasets, revealing that MaMMUT (combining contrastive and captioning losses) consistently outperforms CLIP at compute scales above 10^10-10^11 GFLOPs across three open datasets (DataComp-1.4B, Re-LAION-1.4B, DFN-1.4B) despite CLIP's advantages at smaller scales, while DFN-1.4B emerges as the superior pre-training dataset and constant learning rate schedules enable 98% compute reduction for scaling law studies without compromising trend consistency, with all models, intermediate checkpoints, and training code released openly alongside openMaMMUT-L/14 achieving 80.3% zero-shot ImageNet-1k accuracy, establishing a reproducible framework that challenges single-point model comparisons and validates architectural claims through comprehensive scalability analysis rather than anecdotal performance metrics.
Key insights
- Models can exhibit performance crossovers where architectures appearing inferior at small scales become superior at large scales, as demonstrated by MaMMUT consistently overtaking CLIP around 10^10-10^11 GFLOPs
- Constant learning rate schedules can derive consistent scaling law trends with 98% less compute than cosine schedules, making scaling studies more accessible
- Dataset quality significantly impacts scalability, with DFN-1.4B showing superior performance across architectures compared to DataComp-1.4B and Re-LAION-1.4B
- Architectural claims like SigLIP's superiority over CLIP may not hold under controlled comparisons with identical training data and compute budgets
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.