Reproduce “Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning” on OpenResearch

About this paper

Xiaokun Wang, Peiyu Wang, Jiangbo Pei, Wei Shen, Yi Peng, Yunzhuo Hao, Weijie Qiu, Ai Jian and 4 more

Skywork-VL Reward offers an effective multimodal reward model for evaluating vision-language model (VLM) outputs across diverse tasks, including complex reasoning. It achieves 73.1% overall accuracy on VL-RewardBench and enhances VLM reasoner performance on MathVista from 69.2% to 73.5% when integrated into Mixed Preference Optimization.

Key insights

• A meticulously curated, large-scale multimodal preference dataset, combining diverse open-source data with targeted in-house human annotations for complex reasoning, is crucial for training effective multimodal reward models.
• Adapting a strong pre-trained VLM (Qwen2.5-VL-7B-Instruct) by replacing its generative head with a discriminative reward head, coupled with partial parameter freezing, enables efficient and effective learning of multimodal preferences.
• A robust multimodal reward model can not only accurately evaluate VLM outputs but also serve as a powerful signal source for advanced alignment techniques like Mixed Preference Optimization, leading to direct improvements in VLM reasoning capabilities.

Summary by alphaXiv.