Reproduce “EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhuofan Zong, Dongzhi Jiang, Bingqi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, Hongsheng Li
EasyRef, from CUHK MMLab, SenseTime Research, and Shanghai AI Lab, introduces a plug-and-play method that enables diffusion models to generate images conditioned on multiple reference images and a text prompt. It leverages a Multimodal Large Language Model to effectively identify and synthesize consistent visual elements across diverse inputs, achieving superior zero-shot multi-reference generation and establishing the new MRBench benchmark.
Key insights
- MLLMs can effectively comprehend and aggregate consistent visual elements from multiple diverse reference images, providing a more sophisticated conditioning mechanism than traditional embedding averaging.
- A progressive training scheme across alignment, single-reference, and multi-reference stages is crucial for achieving robust zero-shot generalization and preserving fine-grained details.
- Introducing learnable reference tokens and processing them at the MLLM's final layer offers an efficient way to handle long multimodal contexts without sacrificing performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.