Reproduce “Concept-skill Transferability-based Data Selection for Large Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jaewoo Lee, Boyang Li, Sung Ju Hwang

COINCIDE introduces an efficient visual instruction tuning data selection method for Large Vision-Language Models, leveraging concept-skill transferability. It achieves 97.4% performance relative to full data training using only 20% of the LLaVA-1.5 dataset, and surprisingly outperforms full dataset training by 1.0% on Vision-Flan with 16.7% of the data, significantly reducing training time.

Key insights

  • Multimodal neuron activations from different intermediate layers of an LVLM encode distinct visual-linguistic concepts and skills, which can be combined to represent complex concept-skill compositions.
  • The transferability of knowledge between concept-skill compositions can be efficiently approximated by the cosine similarity of their cluster centroids in the feature space.
  • Achieving broad generalization in LVLMs requires data selection that explicitly prioritizes diverse "concept-skill compositions," rather than relying on single-score metrics that often lead to biased data subsets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.