Reproduce “Comparison Visual Instruction Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wei Lin, Muhammad Jehanzeb Mirza, Sivan Doveh, Rogerio Feris, Raja Giryes, Sepp Hochreiter, Leonid Karlinsky

Researchers developed a two-phase data generation and tuning approach, CaD-VI, to enhance Large Multimodal Models (LMMs) in identifying and describing both commonalities and differences between images. This work introduced new datasets, CaD-Inst and CaD-QA, and demonstrated improved performance on various visual question answering benchmarks and open-ended comparative tasks, showing up to a 17.5% gain on some specific benchmarks.

Key insights

  • A scalable methodology for generating structured commonality and difference visual instruction data can be achieved by first leveraging LLMs with dense captions, then iteratively improving data generation with an LMM tuned on this initial data.
  • Explicitly tuning LMMs with structured CaD instructions significantly improves their ability to perform comparative visual analysis and generalizes to other complex visual reasoning tasks.
  • The inclusion of commonality summaries, difference summaries, and text-to-image retrieval tasks during training contributes to cumulative performance improvements across various benchmarks while preserving general visual-language capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.