Reproduce “MultiRef: Controllable Image Generation with Multiple Visual References” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoxi Chen, Dongping Chen, Siyuan Wu, Sinan Wang, Shiyun Lang, Petr Sushko, Gaoyang Jiang, Yao Wan and 1 more
A new benchmark and synthetic data engine were established to evaluate and facilitate research in generating images from multiple visual references. This framework revealed that current advanced models struggle to integrate diverse visual inputs, with the top-performing model showing substantial performance drops compared to its single-reference capabilities.
Key insights
- State-of-the-art generative models, regardless of whether they are unified or compositional frameworks, consistently struggle to effectively integrate information from multiple visual references.
- The way conditioning inputs are formatted (e.g., bounding box representations, depth map styles) and the order in which multiple references are presented significantly impact generation performance in a model-specific manner.
- While textual captions generally improve depth fidelity and aesthetic quality, they can sometimes lead to decreased performance for precise structural references, suggesting potential conflicts or over-constraining effects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.