Reproduce “MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark” on OpenResearch

About this paper

Haiyang Guo, Fei Zhu, Hongbo Zhao, Fanhu Zeng, Wenzhuo Liu, Shijie Ma, Da-Han Wang, Xu-Yao Zhang

Researchers introduced MCITlib, an open-source library and benchmark for Multimodal Continual Instruction Tuning (MCIT) of Multimodal Large Language Models (MLLMs), providing a standardized platform to evaluate eight existing MCIT algorithms across three new benchmarks. Experiments conducted with MCITlib showed that current MCIT methods partially mitigate forgetting on learned tasks but often degrade the MLLMs' general-purpose capabilities, as indicated by a consistent decline in performance on general multimodal benchmarks.

Key insights

• Current MCIT methods demonstrate a partial ability to mitigate catastrophic forgetting on the specific downstream tasks they are trained on.
• A critical challenge identified is that existing MCIT methods often degrade the MLLM's original, broader generalization abilities acquired during pre-training.
• The lack of a unified, standardized platform significantly impeded reproducible and comparable research in Multimodal Continual Learning (MCL).

Summary by alphaXiv.