Reproduce “Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nilay Naharas, Dang Nguyen, Nesihan Bulut, Mohammadhossein Bateni, Vahab Mirrokni, Baharan Mirzasoleiman
Researchers from UCLA and Google Research developed XMAS, a data selection method for fine-tuning Large Vision-Language Models (LVLMs) that effectively identifies non-redundant data by analyzing cross-modal alignment trajectories. This approach reduces training data by 50% for LLaVA-665k and 85% for Vision-Flan, preserving full model performance and leading to a 1.2x overall training speedup.
Key insights
- Cross-modal attention matrices and their singular values directly correlate with example-specific gradient similarity in LVLMs, offering a proxy for redundancy.
- Tracking the evolution of cross-modal alignment scores across training checkpoints (alignment trajectories) effectively captures how examples influence model learning over time.
- Utilizing a smaller proxy LVLM allows for efficient computation of these alignment trajectories, which are transferable to larger target LVLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.