Reproduce “GIVT: Generative Infinite-Vocabulary Transformers” on OpenResearch

About this paper

Michael Tschannen, Cian Eastwood, Fabian Mentzer

GIVT generates class-conditional images by modeling continuous image latents with a decoder-only transformer, avoiding codebook tokens. A variational autoencoder encodes and decodes images, while GIVT projects latent vectors into transformer states and predicts Gaussian-mixture distributions. On 512×512 ImageNet, it reports FID 4.86, a distance between generated and reference image-feature distributions, versus MaskGIT’s 7.80 at 16 steps; this compares pipelines, not quantization alone.

Key insights

• Directly modeling continuous, real-valued latent sequences with a transformer decoder is feasible and can surpass the performance of existing discrete tokenization methods for generative tasks.
• A more regularized latent space from the β-VAE, even at the cost of some reconstruction fidelity, is beneficial for the downstream GIVT's generative performance and easier to model.
• Employing a Gaussian Mixture Model (GMM) with multiple components for predicting continuous distributions and utilizing advanced sampling strategies like DB-CFG are critical for achieving high-fidelity image generation with GIVT.

Summary by alphaXiv.