Reproduce “Autoregressive Pre-Training on Pixels and Texts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yekun Chai, Qingyi Liu, Jingwu Xiao, Shuohuan Wang, Yu Sun, Hua Wu

The paper introduces PixelGPT, an autoregressive model pre-trained directly on 24-bit RGB pixel representations of text, and DualGPT, which integrates both pixel and tokenized text modalities. This work demonstrates that tokenization-free pixel-based models achieve competitive language understanding, particularly in multilingual contexts where they show substantial gains over token-based methods in distant languages, and that full-color visual information enhances semantic comprehension.

Key insights

  • Autoregressive pre-training on raw RGB pixel data is a viable and robust approach for language understanding, demonstrating competitive performance against state-of-the-art token-based and existing pixel-based models.
  • Pixel-based models effectively overcome the "vocabulary bottleneck" and exhibit enhanced cross-lingual generalization, particularly in linguistically distant languages, by learning from the universal visual forms of text.
  • Multimodal pre-training, especially with explicitly paired image-text data, leads to synergistic benefits that significantly improve overall language understanding performance and multilingual interpretive proficiency.
  • Utilizing full 24-bit RGB images is crucial for capturing rich visual cues (e.g., text color, highlighting, full-color emojis) that meaningfully contribute to semantic understanding, outperforming grayscale representations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.