Reproduce “Towards Lossless Ultimate Vision Token Compression for VLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dehua Zheng, Mouxiao Huang, Borui Jiang, Hailin Hu, Xinghao Chen
A framework called Lossless Ultimate Vision Token Compression (LUVC) was developed by Huawei Noah’s Ark Lab to enhance Visual Language Model (VLM) inference efficiency, achieving approximately a 2x speedup with negligible accuracy degradation. The training-free approach enables immediate deployment across various VLM architectures and tasks.
Key insights
- Low-frequency visual tokens inherently carry more global and semantically crucial information for Transformers, making them robust and suitable for retention, while high-frequency components often represent noise and can be effectively pruned.
- Maintaining the 2D spatial structure of visual tokens during compression is critical for compatibility with prevalent 2D projector architectures and for preserving vital spatial information.
- Frequency domain analysis provides a robust, attention/similarity-free mechanism for identifying and pruning redundant visual tokens, overcoming limitations like position bias and incompatibility with modern attention optimizations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.