Reproduce “LLaVA-KD: A Framework of Distilling Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue and 2 more
A new framework called LLaVA-KD facilitates the transfer of complex knowledge from large to small Multimodal Large Language Models (MLLMs), enabling resource-efficient models to achieve enhanced multimodal understanding. It combines novel distillation strategies for visual and linguistic tokens with a three-stage training paradigm, improving average performance by up to 5.3% on standard benchmarks for 1B and 2B parameter models.
Key insights
- Explicitly distilling both visual and linguistic token distributions (MDist) and visual token relationships (RDist) is crucial for effective knowledge transfer in multimodal contexts.
- A structured, multi-stage training approach (DPT-SFT-DFT) significantly outperforms conventional two-stage training by progressively aligning visual-linguistic representations and refining multimodal understanding.
- An optimal capacity match between teacher and student models is important for effective knowledge transfer, as excessively large teacher models can sometimes hinder distillation success.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.