Reproduce “MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaojun Shan, Qi Cao, Xing Han, Haofei Yu, Paul Pu Liang

MINT proposes a new instruction tuning strategy for large multimodal models that categorizes tasks by their underlying multimodal interaction patterns (redundancy, uniqueness, synergy). This approach leads to substantial improvements in model performance and generalization, with gains up to 26.7% on specific task categories, while outperforming larger, state-of-the-art LMMs.

Key insights

  • Grouping multimodal tasks based on dominant interaction types (redundancy, uniqueness, synergy) fosters transferrable skill learning within groups, effectively mitigating negative interference from mismatched tasks.
  • Quantifying multimodal interactions at a dataset level provides a principled metric for task similarity, enabling more effective and structured curriculum learning for LMMs beyond superficial cues.
  • Strategic fine-tuning guided by interaction-aware task grouping can significantly enhance LMM performance and generalization without relying solely on increased model size or complex architectural modifications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.