Reproduce “UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, Dan Xu

The UNIGP framework introduces a Diffusion Transformer capable of simultaneously performing high-quality controllable image and geometry generation, alongside accurate dense perception tasks like depth and surface normal estimation. It achieves this by modeling the joint distribution of RGB, depth, and surface normal maps, outperforming prior unified models and achieving competitive results against specialized state-of-the-art methods across various benchmarks.

Key insights

  • Disentangling the gradient flow within the DUGP branch is crucial for preserving the generative priors of the frozen backbone while allowing effective learning of dense prediction tasks.
  • Joint training of generation and perception tasks leads to complementary benefits: perception training enhances structural alignment in generation, and generative priors improve fine-grained detail and generalization in perception.
  • Modeling the joint distribution of different modalities (RGB, depth, normal) within a single DiT enables a truly versatile system capable of both understanding and creating visual content with geometric consistency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.