Reproduce “Emerging Properties in Unified Multimodal Pretraining” on OpenResearch

About this paper

Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu and 4 more

BAGEL is a unified model for image understanding, generation, and complex image editing. It trains on interleaved text, image, and video sequences, while understanding and generation experts share attention over multimodal context. On IntelligentBench, BAGEL scores 44.9 versus 14.9 for Step1X-Edit. It reaches 85% of peak at 3.61T tokens, later than conventional tasks; training also changes resolution and data mix.

Key insights

• Complex compositional reasoning and 'intelligent editing' capabilities emerge non-linearly and significantly later during large-scale pretraining (beyond 2.6T tokens) compared to basic understanding and generation.
• The unified Mixture-of-Transformer-Experts architecture with shared self-attention effectively supports both multimodal understanding and generation without introducing heuristic bottlenecks, outperforming dense and standard MoE designs.
• The inclusion of visual-semantic reasoning provided by ViT tokens is critical for complex, intelligence-demanding tasks like 'Intelligent Editing,' while having less impact on basic editing.
• Explicit Chain-of-Thought (CoT) reasoning, integrated into both generative and understanding tasks, substantially boosts the model's ability to leverage world knowledge and follow complex instructions, improving scores by up to 0.18 points on WISE.

Summary by alphaXiv.