Reproduce “[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding

VTC-CLS introduces a training-free method for efficient Multimodal Large Language Models (MLLMs) by compressing visual tokens based on the [CLS] token's attention scores from the visual encoder. This approach achieved state-of-the-art performance in visual-language tasks, accelerating inference by up to 1.7x while effectively preserving model capabilities.

Key insights

  • The [CLS] token within the visual encoder inherently contains valuable prior knowledge about the importance of visual tokens for MLLM operations.
  • There is a strong empirical consistency and high Spearman's rank correlation between the visual encoder's [CLS] attention and the LLM's output token attention, validating [CLS] as a reliable importance proxy.
  • Aggregating [CLS] attention scores from multiple visual encoder layers provides a more robust and comprehensive importance assessment than relying on a single layer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.