Reproduce “Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start” on OpenResearch

About this paper

Kun Chen, Peng Shi, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao, Lin Ma

The METIS-SPECS framework introduces a self-distilled preference-based cold start for Vision-Language Models, decoupling surface-form learning from deep reasoning to improve downstream reinforcement learning. This approach enhances generalization and achieves notable performance gains, including a 4.1% improvement on MEGA-Bench Core and 12.2% on MathVista.

Key insights

• Instruction-style overfitting from SFT-based cold starts limits VLM generalization, whereas preference-based training (DPO) offers superior out-of-distribution capabilities.
• Decoupling shallow learning objectives (e.g., output format) during the cold start from deep reasoning in the subsequent RL phase leads to more stable, efficient, and higher-performing VLM training.
• Self-distilled preference data generated by an exploratory VLM is more effective than data from external large models, proving crucial for tailored VLM cold start alignment.

Summary by alphaXiv.