Reproduce “NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation” on OpenResearch

About this paper

Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, Michael Qizhe Shieh

NoisyRollout introduces a data augmentation method for training Vision-Language Models (VLMs) using Reinforcement Learning, applying controlled visual noise during RL rollouts to enhance reasoning capabilities. This approach improves robustness and generalization, achieving superior performance on five out-of-domain benchmarks and demonstrating exceptional sample efficiency compared to existing methods.

Key insights

• Injecting controlled visual noise during RL rollouts creates targeted perceptual diversity, leading to more robust reasoning paths and improved generalization.
• Reward differences between clean and noisy rollouts provide implicit contrastive signals, which help refine the model's visual perception by reducing negative exploration space.
• An annealing schedule for noise strength is critical for stable training, enabling a smooth transition from broad exploration early on to more 'on-policy' learning as the VLM's capabilities improve.

Summary by alphaXiv.