Reproduce “Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bohan Wang, Zhongqi Yue, Fengda Zhang, Shuo Chen, Li'an Bi, Junzhe Zhang, Xue Song, Kennard Yanting Chan and 10 more
Huawei's Media Technology Institute developed Selftok, a discrete visual tokenizer that represents images as autoregressive tokens derived from the reverse diffusion process. This approach facilitates purely discrete Vision-Language Models (VLMs) and enables effective visual Reinforcement Learning (RL), achieving state-of-the-art visual generation and reasoning capabilities.
Key insights
- The reverse diffusion process exhibits an inherent recursive structure that can be directly mapped to an autoregressive causal graph, enabling the learning of naturally autoregressive visual tokens.
- Autoregressive discrete visual tokens, unlike conventional spatial tokens, adhere to the Bellman equation, which is foundational for enabling optimal policy improvement in Reinforcement Learning for visual generation.
- Discarding the spatial prior in visual tokenization and adopting an autoregressive structure allows for a minimalist, unified LLM-centric architecture for multimodal AI, simplifying integration and training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.