Reproduce “Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lin Chen, Xiaoke Zhao, Kun Ding, Weiwei Feng, Changtao Miao, Zili Wang, Wenxuan Guo, Ying Wang and 4 more
Align-TI, a knowledge distillation framework from Ant Group and academic collaborators, effectively compresses multimodal large language models by aligning dynamic token interactions rather than static next-token predictions. The resulting Align-TI-2B model outperforms larger MLLMs like LLaVA-1.5-7B by 7.0% and MobileVLM-V2-7B by 2.2%, while significantly improving inference speed and memory efficiency.
Key insights
- Effectively distilling MLLMs requires moving beyond static next-token alignment to capture dynamic token interactions during both vision-instruction processing and autoregressive generation.
- Quantifying and leveraging the teacher's instruction-aware visual focus in specific 'middle-deep' layers significantly enhances the student's ability to extract salient visual information.
- Aligning dynamic token transition probabilities, estimated via on-policy sampling and parallelized computation, is critical for mitigating exposure bias and improving the coherence of generated sequences in distilled MLLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.