Reproduce “ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer” on OpenResearch

About this paper

Zhen Han, Zeyinzi Jiang, Yulin Pan, Jingfeng Zhang, Chaojie Mao, Chenwei Xie, Yu Liu, Jingren Zhou

ACE, an all-round creator and editor from Tongyi Lab, integrates a Diffusion Transformer with a unified input paradigm to perform a wide array of visual generation and editing tasks. The model demonstrates competitive to superior performance across various benchmarks, notably achieving the highest prompt following scores in 7 out of 12 global and 8 out of 10 local editing tasks in human evaluations, and recording 70.07% Face Similarity in facial editing.

Key insights

• A unified Diffusion Transformer architecture, augmented with condition tokenizing, image indicator embeddings, and a long-context attention block, effectively unifies diverse multi-modal inputs and enables comprehensive instruction-guided visual generation and editing.
• The integration of image indicator embeddings implicitly associates textual instructions with specific visual inputs, improving the model's ability to interpret and act upon complex multi-modal queries.
• Scalable and high-quality data generation through a hybrid approach (synthesis from expert models and mining from massive databases with MLLM-driven instruction labeling) is crucial for training generalist visual foundation models that generalize across numerous tasks.

Summary by alphaXiv.