Reproduce “CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers” on OpenResearch

About this paper

Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, Jiaqi Wang

Researchers from Tsinghua University and Shanghai AI Laboratory introduce CrossGET, a framework to accelerate vision-language Transformers by adaptively ensembling tokens during inference. It substantially reduces computational costs, achieving up to a 93% increase in throughput and 86% GFLOPs reduction across various multimodal tasks, including image-text retrieval and visual question answering, with negligible performance impact.

Key insights

• Cross-modal information, encapsulated by specially trained "cross tokens," can effectively guide token reduction across diverse VLM architectures, including both modality-independent and modality-dependent models.
• A non-iterative complete-graph soft matching approach ensures efficient and parallel token grouping, which is crucial for real-time inference acceleration.
• The framework's orthogonality to other acceleration techniques (pruning, quantization) allows for cumulative efficiency gains and shows that accelerating larger models can achieve superior performance-cost trade-offs compared to smaller unaccelerated models.

Summary by alphaXiv.