Reproduce “CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuliang Cai, Mohammad Rostami
CluMo introduces a cluster-based modality fusion prompt for Vision-Language Models, enabling them to continually learn new Visual Question Answering (VQA) tasks without catastrophic forgetting. The method achieves state-of-the-art accuracy and lower forgetting rates on CLOVE benchmarks by leveraging both visual and textual information for prompt selection.
Key insights
- Explicitly fusing information from both visual and textual modalities for prompt selection is crucial for effective continual learning in multimodal tasks like VQA.
- Cluster-based training of modality-specific prompt keys ensures they are semantically representative, leading to more precise prompt selection and improved generalization.
- Knowledge distillation effectively stabilizes learning and mitigates catastrophic forgetting, complementing the prompt-based task adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.