Reproduce “Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction” on OpenResearch

About this paper

Rui Fonseca, Bruno Martins, Gil Rocha

TOMCap, developed by researchers at INESC-ID, Instituto Superior Técnico, University of Lisbon, is a text-only training framework for image captioning that leverages retrieval augmentation and a modality gap correction mechanism for CLIP embeddings. It achieves competitive performance against fully supervised models and outperforms previous text-only methods, demonstrating strong generalization capabilities on out-of-domain datasets like NoCaps.

Key insights

• Modality gap correction, specifically aligning mean and standard deviation of embeddings, is critical for bridging the gap between text-only training and image-based inference, significantly improving performance (e.g., from 79.3 to 103.4 CIDEr on MSCOCO).
• Retrieval augmentation and direct conditioning on image embeddings are complementary; the full TOMCap model, combining both, achieves superior performance compared to using either method in isolation.
• Training the model to predict the closest retrieved caption rather than a ground-truth caption, combined with LoRA for efficient fine-tuning, effectively leverages pre-trained language model knowledge and improves generalization.

Summary by alphaXiv.