Reproduce “Continual Learning for Multiple Modalities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyundong Jin, Eunwoo Kim

A framework named COMM is introduced for continual learning across diverse modalities like image, video, audio, depth, and text by adapting pre-trained vision-language models through prompts. It consistently achieved superior performance compared to existing continual learning methods, exhibiting higher average incremental accuracy and reduced forgetting in multimodal sequential learning.

Key insights

  • Effectively mitigating catastrophic forgetting in multimodal continual learning requires strategies to manage both intra-modal knowledge evolution and inter-modal interference.
  • Prompt-based adaptation of pre-trained vision-language models offers a robust and efficient way to integrate new modalities sequentially without altering the large backbone, benefiting from existing semantic alignments.
  • Selectively integrating knowledge from previously learned modalities based on relevance to the current input significantly reduces destructive cross-modal interference.
  • Periodically re-aligning modality embeddings with their associated text embeddings helps restore semantic consistency and counter bias towards recently learned tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.