Reproduce “ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, Jingren Zhou
Researchers from Tongyi Lab developed ACE++, an instruction-based diffusion framework that enhances image creation and editing by introducing LCU++ for efficient multimodal input handling and a two-stage training scheme. The framework achieves high image quality and accurate instruction following across diverse tasks, demonstrating robust subject and portrait consistency, as well as versatile local and general editing capabilities.
Key insights
- Concatenating conditional context (image, mask) with noisy latents in the channel dimension for "0-ref tasks" (LCU++) avoids disrupting the pre-trained attention mechanism of the base T2I model, leading to more efficient fine-tuning.
- A dual training objective (L_ref + L_tar) explicitly guides the model to learn context-aware generation capabilities, ensuring coherence with provided reference images and target outputs.
- The two-stage training scheme effectively minimizes fine-tuning effort and computational cost by building progressively from single-condition adaptation to multi-condition mastery, maximizing the benefits of pre-trained models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.