Reproduce “Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu
Researchers from Zhejiang University and Ant Group developed Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that enables Vision-Language Models to jointly learn answer generation and self-verification. This method achieves significant performance gains in best-of-N selection across multimodal math reasoning, visual grounding, and mobile agent tasks, while reducing inference latency by up to 53.5% compared to dual-model approaches.
Key insights
- Reframing self-verification as a preference ranking task, rather than absolute correctness, provides more stable and informative gradient signals for training unified policies.
- Decoupling the advantage signals for generation and verification objectives, combined with token-level masking, is crucial to prevent reward hacking and enable synergistic learning.
- A single VLM can be efficiently trained to perform both robust answer generation and calibrated self-assessment, overcoming the computational and deployment overhead of multi-model systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.