Reproduce “DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa and 1 more
DREAM introduces a speculative decoding framework tailored for Vision-Language Models (VLMs), incorporating a cross-attention mechanism, entropy-adaptive intermediate feature selection, and visual token compression. The method achieves a 1.5x to 3.6x speedup over conventional autoregressive decoding across various VLMs and multimodal benchmarks while maintaining output quality.
Key insights
- Speculative decoding can be effectively extended to multimodal models by incorporating adaptive mechanisms for knowledge transfer and efficient processing of multimodal inputs.
- Dynamically selecting intermediate features with low attention entropy from the target model provides superior supervision for training a draft model, enhancing its predictive accuracy and token acceptance.
- Intelligent compression of visual tokens based on their importance significantly reduces the computational load on the draft model in VLMs, leading to faster decoding with minimal impact on accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.