Reproduce “SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, M. Salman Asif
A training-free framework, Singular Subspace Alignment and Merging (SSAM), integrates independently trained specialist Multimodal Large Language Models into a unified model capable of processing arbitrary modality combinations. The method consistently surpassed existing baselines, achieving up to 19.03 percentage points higher accuracy on the challenging MCUB-4 benchmark and 14 percentage points on AVQA, without requiring new paired multimodal training data.
Key insights
- Task-specific knowledge within specialist MLLMs can be represented as low-rank parameter update vectors in a shared language decoder space.
- Identifying a shared, low-rank consensus subspace across these parameter update vectors allows for effective alignment and mitigation of interference when merging heterogeneous MLLMs.
- Projecting and averaging updates within this consensus subspace integrates complementary knowledge, enabling a unified MLLM to handle arbitrary modality combinations without additional training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.