Reproduce “MobileLLM-Pro Technical Report” on OpenResearch

About this paper

Patrick Huber, Ernie Chang, Wei Wen, Igor Fedorov, Tarek Elgamal, Hanxian Huang, Naveen Suda, Chinnadhurai Sankar and 11 more

Meta Reality Labs introduces MobileLLM-Pro, a 1-billion-parameter large language model designed for efficient on-device deployment, achieving state-of-the-art performance across 11 benchmarks and supporting context windows up to 128,000 tokens. The model maintains strong capabilities even under 4-bit quantization, enabling practical inference on resource-constrained devices.

Key insights

• Dynamically guided data mixing based on an offline simulation of data utility significantly enhances small model training robustness and overall performance.
• Long-context capabilities can be effectively extended in compact models through implicit positional distillation, inheriting knowledge from larger models without direct long-context data training.
• Combining diverse capabilities in small models is achievable by training specialist branches and merging them via non-uniform weight averaging, capturing emergent specializations.
• Quantization-Aware Training with self-distillation is critical for mitigating performance degradation in 4-bit quantized models, leading to significantly better results than post-training quantization.

Summary by alphaXiv.