Reproduce “Image Difference Captioning with Pre-training and Contrastive Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Linli Yao, Weiying Wang, Qin Jin

Researchers from Renmin University of China developed a pre-training-finetuning framework for Image Difference Captioning (IDC) that leverages contrastive learning and cross-task data. The approach achieved new state-of-the-art results, boosting CIDEr on CLEVR-Change from 118.7 to 128.9 and ROUGE-L on Birds-to-Words from 45.6 to 49.1.

Key insights

  • Self-supervised pre-training, particularly through Masked Visual Contrastive Learning and Fine-grained Difference Aligning, is critical for learning discriminative visual and fine-grained cross-modal difference representations.
  • A dedicated Image Difference Encoder module is essential for effectively identifying and processing subtle visual discrepancies between two input images.
  • Expanding training data with general image captioning and fine-grained visual classification datasets significantly improves model robustness and performance by leveraging diverse forms of background knowledge.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.