Reproduce “Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment” on OpenResearch

About this paper

Xing Xie, Jiawei Liu, Ziyue Lin, Huijie Fan, Zhi Han, Yandong Tang, Liangqiong Qu

The Autoregressive Representation Alignment (ARRA) framework redefines the training objective for large language models (LLMs) to achieve global coherence in text-to-image generation without altering their architecture. This approach reduced FID by 16.6% on ImageNet and 25.5% on MIMIC-CXR, demonstrating enhanced visual quality and semantic consistency.

Key insights

• Global coherence in text-to-image generation can be achieved through a refined training objective and representation alignment, rather than through complex architectural modifications of LLMs.
• Aligning representations at earlier LLM layers provides sufficient global semantic guidance, enabling deeper layers to focus on capturing high-frequency visual details more effectively.
• The <HYBNEXT> token provides an effective mechanism for continuously injecting global visual constraints throughout the autoregressive generation process, bridging local token prediction with global understanding.

Summary by alphaXiv.