Reproduce “SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning” on OpenResearch

About this paper

Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, Sam Kwong

SEFE introduces a framework to mitigate catastrophic forgetting in Multimodal Continual Instruction Tuning by differentiating between superficial and essential forgetting. The method combines Answer Style Diversification and RegLoRA, achieving top performance on the CoIN benchmark by addressing both response style consistency and factual knowledge retention.

Key insights

• Catastrophic forgetting in MLLMs can be effectively decomposed into two distinct types: 'superficial forgetting' (deviation in answer style) and 'essential forgetting' (loss of factual knowledge).
• Diversifying answer styles during training not only mitigates superficial forgetting but also indirectly helps alleviate essential forgetting by reducing inter-task differences in output formats.
• Targeted regularization of specific, critical parameters in LoRA weight updates allows for effective preservation of prior knowledge while maintaining flexibility for learning new information, outperforming uniform regularization approaches.

Summary by alphaXiv.