Reproduce “Scaling Laws for Native Multimodal Models” on OpenResearch

About this paper

Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, Alaaeldin El-Nouby

Researchers at Apple conducted a comprehensive study on scaling laws for native multimodal models, demonstrating that early-fusion architectures trained from scratch can achieve comparable performance to late-fusion approaches while offering superior training efficiency and lower deployment costs. Integrating Mixture of Experts into these early-fusion designs further enhances performance and enables implicit specialization to different modalities.

Key insights

• Early-fusion architectures trained from scratch yield similar validation loss scaling exponents compared to late-fusion architectures, suggesting no inherent performance advantage for late-fusion in a from-scratch training regime.
• Native Multimodal Models, particularly early-fusion variants, demonstrate superior training efficiency (faster training, less memory) and more favorable parameter-to-data trade-offs for compute-optimal scaling, leading to lower inference costs.
• Mixture of Experts (MoEs) significantly improve the performance of early-fusion NMMs at the same inference cost, and their experts implicitly learn to specialize in different modalities (image vs. text) without explicit modality-aware routing.

Summary by alphaXiv.