Reproduce “Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Evelyn Chee, Wynne Hsu, Mong Li Lee

Researchers at the National University of Singapore developed a framework for multi-modal continual learning, employing cross-modality adapters and tailored loss functions to mitigate catastrophic forgetting. This approach improved classification accuracy on audio-visual (AVE) by 3.40% to 91.91% and visual-inertial (UESTC-MMEA) datasets by 1.85% to 95.54% in class-incremental settings.

Key insights

  • Cross-modality adapters with a Mixture-of-Experts architecture effectively capture dynamic interactions between modalities, outperforming independent modality-specific adaptation in continual learning.
  • Aligning each modality's representation with a fused joint representation, as opposed to direct inter-modality alignment, helps preserve crucial modality-specific features while improving overall discriminability.
  • A combination of freezing robust experts and specific knowledge preservation losses is essential for retaining prior multi-modal knowledge and preventing catastrophic forgetting over sequential tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.