Reproduce “PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu
PivotMerge merges visual-to-language projectors trained on separate image-text data splits, rather than merging multimodal models after fine-tuning. It maps projector updates into a shared basis, separates alignment cores from residuals, filters inconsistent residual directions, and weights cores by layer using held-out examples. On clustered CC12M splits, it reaches an eight-benchmark average of 41.8 versus TIES’s 39.6, in the LLaVA-1.5-7B setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.