Reproduce “DreamOmni: Unified Image Generation and Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia

DreamOmni proposes a unified model for image generation and diverse editing tasks, including inpainting, instruction-based editing, and subject-driven generation. The model leverages a hybrid architecture and an innovative synthetic data pipeline to achieve state-of-the-art or competitive performance across these varied capabilities.

Key insights

  • Combining the computational efficiency of Diffusion Transformers (DiT) with the rapid convergence of UNet-like residual connections creates a powerful and efficient architecture for unified vision tasks.
  • Synthetic data generation through a precise 'sticker-like' collage pipeline, augmented with LLM-refined natural language instructions, effectively addresses the long-standing data scarcity problem for complex image editing tasks.
  • Jointly training a single model on a diverse range of image generation and editing tasks leads to synergistic learning, resulting in improved instruction following, consistency, and overall quality across all functionalities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.