Reproduce “Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pengcheng Wang, Zhiquan Wang, Jayoung Lee, Zhuoyan Xu, Ran Xu, Saurabh Bagchi, Yin Li, Somali Chaterji

SmartVL presents a unified adaptive inference framework for Multimodal Large Language Models (MLLMs), dynamically adjusting visual token count, model depth, and model width based on input content and a given compute budget. This joint adaptation approach achieves a more favorable accuracy-efficiency trade-off across various vision-language tasks, maintaining or improving performance while significantly reducing computational cost.

Key insights

  • Optimal computational allocation in MLLMs necessitates joint, content- and budget-aware control over visual token count, model depth, and model width, rather than independent optimization.
  • A differentiable latency estimator and an asymmetric budget violation loss are crucial for stable end-to-end training across the adaptive, compound discrete search space of MLLMs.
  • While reducing visual token count and model depth offers substantial efficiency gains, preserving full-width attention capacity in the LLM's attention heads is generally important for maintaining inference quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.