Reproduce “Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, Wei Liu
Researchers from Tsinghua University, Peng Cheng Laboratory, Tencent Technology, Tencent Data Platform, and the University of Oxford introduce "verbose images," imperceptibly perturbed visual inputs that compel Large Vision-Language Models (VLMs) to generate significantly longer text sequences. This manipulation increases VLM inference energy consumption and latency by up to 8.56 times compared to original images, highlighting an availability vulnerability.
Key insights
- An empirical study confirmed an approximately linear relationship between the length of sequences generated by VLMs and their associated energy consumption and latency time.
- Verbose images caused VLMs to disperse their attention more uniformly across the entire image, rather than focusing on specific objects, contributing to the generation of longer, less focused sequences.
- The perturbed inputs induced higher rates of object hallucination in VLM outputs, where models described objects not present in the image, further contributing to increased sequence length and complexity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.