Reproduce “UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhipeng Bao, Zhen Zhu, Nupur Kumari, Anurag Bagchi, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert
Carnegie Mellon, UIUC, and Toyota Research Institute's UniGen-AR framework unifies over 15 visual generation tasks, from text-to-image to restoration, by combining a multimodal large language model encoder with an efficient visual auto-regressive decoder. This approach achieves up to 19x faster inference than diffusion-based methods while delivering competitive or superior visual quality.
Key insights
- Visual Auto-Regressive (VAR) models offer a significantly more favorable latency-quality trade-off for Unified Visual Generation compared to diffusion models, demonstrating up to 19x faster inference.
- The specific design of the VQ-VAE tokenizer, including its codebook size and hierarchical structure, is a critical factor for scaling VAR models to achieve high-fidelity generation in a multi-task context.
- A two-stage training strategy, comprising an initial alignment phase and comprehensive multi-task training, is essential for effectively integrating MLLMs with VAR decoders for diverse UVG capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.