Reproduce “Transferability Between Understanding and Generation in Unified Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiwon Kang, Heeji Yoon, Jaewoo Jung, Jaewon Min, Minkyeong Jeon, Biyeon Hwang, Sangwon Jung, Seungryong Kim
Researchers from KAIST AI and collaborators investigated "transferability" in Unified Multimodal Models (UMMs), finding that knowledge gained from image understanding tasks can enhance corresponding image generation capabilities, particularly in models with shared architectural designs. Their method improved generative tasks like counting, spatial relation, and text generation while better preserving overall image quality compared to direct generative fine-tuning.
Key insights
- Cross-task transferability between understanding and generation exists in UMMs, with its strength highly dependent on architectural design; models with fully shared backbones and unified visual encoders exhibit stronger transfer.
- Training on understanding tasks can effectively improve targeted generative capabilities with significantly better preservation of overall image quality and less distribution shift compared to direct generative fine-tuning.
- The dominant direction and strength of transfer are capability-dependent: "und ">" gen" transfer is stronger for high-level concepts like counting, while "gen ">" und" transfer is more robust for fine-grained details like text rendering.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.