Reproduce “NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints” on OpenResearch

About this paper

Changyao Tian, Hao Li, Gen Luo, Xizhou Zhu, Weijie Su, Hanming Deng, Jinguo Zhu, Jie Shao and 6 more

NaViL trains a native image-language model by pairing a learned visual encoder with a pretrained language model. Its scaling tests find diminishing validation-loss returns from enlarging the encoder when language-model size is fixed, while the selected encoder size rises with language-model size across tested configurations. This empirical sizing trend is not a universal rule or proof of better downstream performance.

Key insights

• Initializing the LLM component with a pre-trained checkpoint significantly accelerates multimodal convergence, requiring over 10 times less data and consistently yielding better zero-shot performance.
• Integrating modality-specific Mixture-of-Experts, particularly with attention experts, drastically improves model convergence, enabling comparable performance with only 1/10 of the training data.
• The optimal size of the visual encoder scales approximately proportionally with the LLM size in log scale, indicating that joint and balanced scaling of both components is crucial for optimal native MLLM performance.

Summary by alphaXiv.