Reproduce “Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling” on OpenResearch

About this paper

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan

DeepSeek-AI's Janus-Pro is a unified multimodal model that enhances both visual understanding and generation by scaling its decoupled visual encoding architecture to 7 billion parameters, optimizing its multi-stage training, and incorporating large-scale synthetic data. The model achieves an 80% accuracy on GenEval for instruction-following image generation and an average of 79.2 on MMBench for multimodal understanding, often surpassing larger state-of-the-art unified and specialized models.

Key insights

• The decoupled visual encoding architecture effectively mitigates the conflict between understanding and generation tasks, proving scalable and robust with increased model capacity and data.
• Optimized training strategies, such as focusing pretraining solely on dense text-to-image data and carefully adjusting supervised fine-tuning data ratios, significantly enhance both computational efficiency and overall model performance.
• Integrating large volumes of high-quality synthetic aesthetic data is crucial for accelerating convergence and producing more stable and aesthetically superior text-to-image outputs.

Summary by alphaXiv.