Reproduce “NativeTok: Native Visual Tokenization for Improved Image Generation” on OpenResearch

About this paper

Bin Wu, Mengqi Huang, Weinan Jia, Zhendong Mao

NativeTok introduces a framework for visual tokenization that explicitly embeds causal dependencies into token sequences during the encoding process. This approach aligns tokenization with the requirements of downstream generative models, leading to enhanced image generation performance as measured by Fréchet Inception Distance across different generative architectures.

Key insights

• Explicitly enforcing causal dependencies during visual tokenization, rather than solely optimizing for reconstruction, is more beneficial for improving the performance of downstream generative models.
• Decoupling the modeling of complex image content context from the causal dependency modeling among tokens allows for joint optimization of both reconstruction quality and token ordering.
• The Hierarchical Native Training strategy provides an effective and computationally efficient method to scale autoregressive tokenization architectures to longer sequence lengths.

Summary by alphaXiv.