Reproduce “Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Antoine Chaffin, Ewa Kijak, Vincent Claveau
Researchers at IRISA enhanced CLIP-guided Reinforcement Learning for image captioning by strategically incorporating ground truth captions, which improved caption distinctiveness and writing quality. This method, employing an adaptive discriminator and reward-weighted teacher forcing, yielded higher image-text retrieval scores and increased caption diversity while stabilizing the learning process.
Key insights
- Ground truth captions remain highly valuable in CLIP-guided RL, serving as effective regularization, exploration guides, and stabilizers for training.
- Adaptive discriminators dynamically learn to distinguish human from machine-generated text, offering more effective regularization against reward hacking and leading to increased caption diversity compared to fixed grammar networks.
- A bidirectional contrastive reward formulation is essential for generating captions that are distinct in both image-to-text and text-to-image retrieval contexts, ensuring captions are uniquely descriptive.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.