Reproduce “Heptapod: Language Modeling on Visual Signals” on OpenResearch

About this paper

Yongxin Zhu, Jiawei Chen, Yuanzhe Chen, Zhuo Chen, Dongya Jia, Jian Cong, Xiaobin Zhuang, Yuping Wang and 1 more

Heptapod introduces "next 2D distribution prediction," a novel generative learning paradigm that enables causal Transformers to learn holistic visual semantics. It achieves state-of-the-art performance for causal autoregressive image generation on ImageNet-1K without relying on classifier-free guidance or semantic tokenizers, demonstrating intrinsic semantic learning from reconstruction-focused visual tokens.

Key insights

• Predicting the entire 2D spatial distribution compels the model to learn holistic global structures and semantics, effectively overcoming the "curse of locality" observed in 1D next-token prediction for images.
• The approach implicitly unifies autoregressive modeling with Masked Autoencoding, combining the strengths of both paradigms by having the causal prefix act as context for predicting all future tokens.
• Reconstruction-focused tokenizers are viable, with continuous VAE tokenizers ultimately yielding superior generative quality due to their better reconstruction fidelity, even with slower initial convergence than discrete VQ-VAE tokenizers.

Summary by alphaXiv.