Reproduce “ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, Bohan Zhuang

ZipVL introduces a unified inference framework for Large Vision-Language Models, dynamically sparsifying tokens with layer-wise adaptive ratios to optimize both prefill and decoding phases. This approach achieved up to 2.3x prefill acceleration and 2.8x decoding throughput with minimal accuracy loss on various image and video benchmarks.

Key insights

  • The optimal ratio of important tokens varies significantly across different attention layers and tasks, necessitating a dynamic and adaptive sparsification strategy rather than fixed ratios.
  • A unified approach that leverages a single mechanism for identifying important tokens can simultaneously optimize both the prefill (computational) and decoding (memory) bottlenecks in LVLMs.
  • Approximate attention scores, derived from a small subset of probe tokens, can effectively guide dynamic token sparsification with negligible overhead, enabling practical implementation within fast attention libraries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.