Reproduce “Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuehui Wang, Xuankun Yang, Wei Shen

Researchers at Shanghai Jiao Tong University developed Entropy-Aware Dense Pruning (EADP), a framework to enhance the efficiency and accuracy of Vision-Language Models by filtering textual noise from dense cross-modal scores and employing submodular optimization for visual token selection. EADP consistently improved performance over existing pruning methods across various VLM architectures and tasks, achieving significant reductions in FLOPs (e.g., 66%) and inference speedups (e.g., 1.75x).

Key insights

  • Textual noise from functional words in prompts causes dispersed, near-uniform visual attention scores across images, drowning out signals from semantically relevant entities, which information entropy can effectively quantify and filter.
  • Visual token pruning is fundamentally a structured compression problem, where standard Top-K selection is suboptimal as it leads to feature fragmentation and redundancy rather than holistic representation.
  • A balanced fusion of global semantic context and fine-grained, denoised dense guidance is crucial for forming a robust instruction relevance map that accurately localizes semantic entities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.