Reproduce “Continual Instruction Tuning for Large Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinghan He, Haiyun Guo, Ming Tang, Jinqiao Wang
The paper systematically investigates continual instruction tuning for Large Multimodal Models (LMMs), establishing the first benchmarks and confirming catastrophic forgetting. It proposes task-similarity-informed regularization and model expansion methods, which consistently enhance anti-forgetting performance and parameter efficiency across various LMM architectures.
Key insights
- LMMs definitively suffer from catastrophic forgetting when sequentially instruction-tuned without specific continual learning mechanisms.
- An initial phase of multi-task instruction tuning provides LMMs with a robust "instruction-following capability," which significantly mitigates subsequent catastrophic forgetting.
- Replay-based methods (e.g., ER) and model expansion methods (EProj) are robust for continual instruction tuning in LMMs, while regularization methods become more effective when the LMM has undergone prior multi-task instruction tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.