Reproduce “ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, Hang Xu

Huawei Noah's Ark Lab introduces ILLUME, a unified multimodal large language model capable of both visual understanding and generation, achieving state-of-the-art performance on various benchmarks with only 15 million pretraining image-text pairs, a four-fold reduction compared to prior methods, through a novel semantic vision tokenizer and a self-enhancing alignment scheme.

Key insights

  • Quantizing images based on semantic features, rather than pixel-level reconstruction, through a custom tokenizer significantly accelerates MLLM training convergence and improves generation quality with less data.
  • Explicitly training an MLLM to self-assess the quality and consistency of its own generated outputs with text descriptions fosters a synergistic relationship between understanding and generation capabilities.
  • High-performance unified MLLMs can be developed with substantially reduced pretraining data by optimizing the vision tokenization process and incorporating self-supervision for multimodal alignment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.