Reproduce “Perception Tokens Enhance Visual Reasoning in Multimodal Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G. Shapiro, Ranjay Krishna

This work by researchers from the University of Washington and Google Research introduces "Perception Tokens" to enable Multimodal Language Models (MLMs) to internally generate and reason about intrinsic visual representations like depth maps and bounding boxes. This approach improves performance on visual reasoning tasks, achieving a 6.4 percentage point boost in relative depth estimation on BLINK and substantial gains in object counting over fine-tuned baselines, even surpassing tool-augmented closed-source models in some cases.

Key insights

  • Multimodal Language Models (MLMs) can be augmented to internally generate and reason over intrinsic visual representations, such as depth maps and bounding boxes, akin to how language models perform chain-of-thought reasoning.
  • Integrating these 'Perception Tokens' enables MLMs to perform deeper visual reasoning, addressing tasks requiring precise spatial or instance-level understanding beyond high-level feature extraction.
  • A training framework combining specialist-to-generalist distillation and curriculum learning is effective for teaching MLMs to predict and utilize these novel visual tokens within their reasoning process.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.