Reproduce “BlueLM-2.5-3B Technical Report” on OpenResearch

About this paper

Baojiao Xiong, Boheng Chen, Chengzhi Wang, Daxiong Luo, Dongsheng Xu, Dongyang Liu, Fan Yang, Fangyuan Li and 53 more

vivo AI Lab developed BlueLM-2.5-3B, a 2.9-billion parameter multimodal large language model designed for efficient deployment on edge devices, which uniquely supports both explicit "thinking" and "non-thinking" modes with controllable reasoning budgets. The model demonstrates competitive performance across over 20 multimodal and text benchmarks, often outperforming larger compact models, while achieving this with 23% less pre-training data than some counterparts.

Key insights

• Explicitly controlling reasoning via a "thinking" mode and token budget, even in a 3B-scale MLLM, enables adaptive intelligence, allowing for both efficient simple tasks and complex multi-step reasoning on edge devices.
• Achieving competitive MLLM performance is possible with substantially reduced pre-training data (23% less than comparable models) by effectively initializing the language model and concentrating on high-quality, reasoning-intensive multimodal data.
• A comprehensive development approach involving meticulous architectural design, incremental multi-stage pre-training, and advanced fine-tuning with reinforcement learning is essential for building highly performant yet compact MLLMs.

Summary by alphaXiv.