Reproduce “Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunpeng Qu, Kaidong Zhang, Yukang Ding, Ying Chen, Jian Wang

Researchers from Tsinghua University and Alibaba Group developed SemTok, a visual tokenizer that converts 2D images into compact, semantically rich 1D discrete token sequences. This framework achieves state-of-the-art image reconstruction, with an rFID of 0.67 at 0.125 BPP, and enables high-quality class-conditional image generation with a gFID of 2.34.

Key insights

  • Explicit semantic alignment constraints, incorporating both spatial distillation and contrastive learning, are crucial for guiding the encoder to produce compact, clustered, and semantically rich 1D representations.
  • A two-stage generative training paradigm, combining diffusion-based pre-training with refinement fine-tuning, is more effective than direct pixel-level optimization for exploring diverse latent spaces and enhancing both semantic understanding and fine-grained fidelity.
  • Non-sequential masked autoregressive modeling for 1D tokens effectively leverages their global semantic complementarity, outperforming causal, sequential prediction in downstream image generation tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.