Reproduce “OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation” on OpenResearch

About this paper

Size Wu, Zhonghua Wu, Zerui Gong, Qingyi Tao, Sheng Jin, Qinyue Li, Wei Li, Chen Change Loy

OpenUni introduces a simple and efficient open-source framework that unifies multimodal understanding and generation by connecting frozen MLLMs with frozen diffusion models via a lightweight connector. This approach achieves competitive performance on various multimodal benchmarks while using fewer activated parameters compared to larger, more complex unified models.

Key insights

• A simple, lightweight architecture leveraging frozen MLLMs and diffusion models with a minimalist connector can achieve competitive multimodal understanding and generation performance.
• Learnable queries effectively transfer knowledge from pre-trained multimodal large language models to diffusion models, instilling generative abilities efficiently.
• The two-stage training strategy, combining large-scale pre-training for alignment and high-quality finetuning for refinement, is effective for unified multimodal systems.

Summary by alphaXiv.