Reproduce “Learning to Describe Differences Between Pairs of Similar Images” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Harsh Jhamtani, Taylor Berg-Kirkpatrick

The research introduces the 'spot-the-diff' task, challenging AI to describe changes between two similar images. Researchers from Carnegie Mellon University created a new benchmark dataset and developed a neural model that effectively identifies and narrates visual differences, achieving improved performance over baselines in metrics like BLEU-4 and CIDEr.

Key insights

  • Explicitly modeling visual salience through a learnable prior over difference clusters significantly enhances the quality of generated descriptions.
  • Latent alignment of individual sentences to specific visual difference regions enables the model to focus generation on relevant changes and improves overall description accuracy.
  • A multi-sentence generation approach is essential for exhaustively covering all salient differences present in an image pair.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.