Reproduce “CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marios Krestenitis, Christos Tzelepis, Konstantinos Ioannidis, Stefanos Vrochidis, Ioannis Kompatsiaris, Georgios Tzimiropoulos, Shaogang Gong, Ioannis Patras
CycleCap is a self-supervised fine-tuning framework for Visual-Language Models (VLMs) that enhances image captioning by mitigating vision-language misalignment. It uses cycle consistency, where a VLM generates a caption and a text-to-image model reconstructs the original image, with the similarity providing a reward for optimizing caption quality. The framework consistently improves captioning performance across various VLMs and benchmarks, including CompreCap, CAPability, CapsBench, and MMHal, outperforming existing state-of-the-art methods in reducing hallucinations and enhancing descriptive detail.
Key insights
- Directly applying cycle consistency as a self-supervised training signal effectively improves VLM captioning by reinforcing visual grounding, bypassing the need for expensive human annotations or complex preference datasets.
- Group Relative Policy Optimization (GRPO) facilitates continuous VLM refinement by exploring multiple caption hypotheses and dynamically adapting to an image-grounded reward signal.
- Robust perceptual and semantic image similarity metrics, such as DreamSim, are crucial for effectively quantifying cycle consistency and guiding the VLM towards generating more accurate and detailed captions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.