Reproduce “Unified Generative and Discriminative Training for Multi-modal Large Language Models” on OpenResearch

About this paper

Wei Chow, Juncheng Li, Qifan Yu, Kaihang Pan, Hao Fei, Zhiqi Ge, Shuai Yang, Siliang Tang and 2 more

The Sugar framework unifies generative and discriminative training paradigms in Multi-modal Large Language Models, enabling a single model to excel at both types of tasks. It achieves new state-of-the-art results on several generative benchmarks requiring cognitive and discrimination abilities, and demonstrates superior performance in complex image-text retrieval compared to existing models.

Key insights

• Explicitly modeling semantic relationships between input samples as a structural constraint on hidden states improves Multi-modal Large Language Model capabilities.
• Dynamic sequence alignment using Dynamic Time Warping (DTW) helps MLLMs capture global semantics and fine details in long, interleaved sequences, addressing limitations of causal LLMs.
• Combining multiple discriminative models (DINOv2, CLIP, BGE) via a 'Triple Kernel' enables more robust and fine-grained semantic differentiation across diverse contexts.
• Joint training with balanced generative and discriminative losses allows a single model to maintain strong performance in both paradigms without significant compromise.

Summary by alphaXiv.