Reproduce “LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning” on OpenResearch

About this paper

Muyao Yuan, Muyan Jiao, Jiangyong Ying, Weizhan Zhang, Yuanhong Zhang, Lan Ma, Yuan Gao, Haipeng Du

LLaVAFlow is an information-theoretic distillation framework that mitigates catastrophic forgetting in Multimodal Large Language Models (MLLMs) during fine-tuning by preserving the latent "alignment flow" of cross-modal representations. It consistently improved performance on both downstream tasks and upstream generalization datasets when integrated with existing fine-tuning methods.

Key insights

• Catastrophic forgetting in MLLMs is strongly linked to the degradation of cross-modal alignment during fine-tuning, which can be implicitly represented as an "alignment flow."
• An information-theoretic distillation approach, utilizing matrix-based Rényi’s α-entropy and mutual information, can effectively transfer essential structural alignment knowledge without rigid mimicry.
• A lightweight, single-layer Alignment Flow Module (AFM) is more effective for learning and transferring alignment knowledge, balancing task adaptation and knowledge retention, compared to over-parameterized alternatives.

Summary by alphaXiv.