Reproduce “CFPO: Counterfactual Policy Optimization for Multimodal Reasoning” on OpenResearch

About this paper

Zhangyuan Yu, Wanran Sun, Guangjing Yang, Xiaohu Wu, Qicheng Lao

Counterfactual Policy Optimization (CFPO) introduces a reinforcement learning framework that enhances causal consistency in Large Vision-Language Models by regularizing policy updates through cross-modal counterfactual interventions at the attention layer. This approach consistently improved multimodal reasoning accuracy by 3.17% to 6.25% over GRPO and DAPO baselines, and 1.32% to 2.13% over perception-aware methods, while effectively mitigating hallucinations and improving visual grounding.

Key insights

• Explicitly verifying the causal necessity of attended information at the representation level, through counterfactual intervention, fundamentally enhances an LVLM's reasoning fidelity and reduces reliance on linguistic priors.
• Replacing high-saliency visual features with averaged image token values is a robust intervention strategy, effectively creating a counterfactual state that isolates causal contributions without disrupting latent space alignment.
• The optimal balance between counterfactual regularization and other policy objectives, including entropy loss, is dependent on the base reinforcement learning algorithm used.

Summary by alphaXiv.