Reproduce “Jodi: Unification of Visual Generation and Understanding via Joint Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifeng Xu, Zhenliang He, Meina Kan, Shiguang Shan, Xilin Chen
Chinese Academy of Sciences researchers develop Jodi, a unified diffusion framework that jointly models image generation and understanding tasks through a single linear diffusion transformer that learns joint distributions across seven visual domains (depth, normal, edges, albedo, segmentation, plus images and captions), introducing a role switch mechanism that randomly assigns each domain as generation target, condition input, or ignored during training to enable joint generation of images with their corresponding labels, multi-conditional generation, and simultaneous understanding tasks while achieving comparable performance to specialist models using significantly less data and computational resources.
Key insights
- Joint modeling of visual generation and understanding mirrors human cognitive processes where creating scenes enhances understanding
- Random role assignment during training enables flexible inference patterns for different task combinations
- Single unified model can achieve performance comparable to specialist models while enabling novel cross-domain applications
- Linear scaling architecture allows incorporation of additional visual domains without computational explosion
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.