Reproduce “Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng
Researchers from The Chinese University of Hong Kong and Tencent Hunyuan Research systematically investigated and addressed the compositional ability gap in Vision-Language Models (VLMs) using ComPABench, a novel diagnostic benchmark. Their proposed RL-Ground training strategy, which incorporates visual captioning and fine-grained progress rewards, improved multimodal compositional task performance to 52.8% for a 7B model and demonstrated robust out-of-distribution generalization.
Key insights
- A substantial 'modality gap' exists, indicating that reasoning abilities acquired from text-only training do not automatically transfer to visual inputs in current Vision-Language Models.
- Standard Supervised Fine-Tuning (SFT) can severely impair compositional abilities through catastrophic forgetting, despite improving performance on individual tasks.
- Reinforcement Learning (RL), specifically with fine-grained progress rewards and explicit visual grounding (e.g., captioning before thinking), demonstrates effectiveness in improving multimodal compositional reasoning and out-of-distribution generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.