Reproduce “Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning” on OpenResearch

About this paper

Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng and 1 more

Researchers from Alibaba and The University of Hong Kong introduce Cross-modal Identity Mapping (CIM), an annotation-free reinforcement learning framework that minimizes information loss during image-to-text conversion. CIM consistently improves fine-grained image captioning for various Large Vision-Language Models, achieving an average 3.7% improvement in CAPTURE over base models and boosting Attributes F1 by 12.3% and Relations QA by 20.4% for Qwen2.5-VL-7B on COCO-LN500.

Key insights

• Empirical verification shows that LVLMs tend to accurately describe coarse concepts but often omit or misrepresent fine-grained details during image-to-text conversion.
• The quality of an image caption, in terms of detail richness and precision, can be quantitatively assessed by measuring the consistency and relevance among images retrieved using that caption.
• Leveraging the complementary roles of GRC (detail richness) and QIR (precision) as an RL reward effectively guides LVLMs to minimize information loss in cross-modal mapping.

Summary by alphaXiv.