Reproduce “FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang and 1 more
FOVEA introduces a cache-friendly approach for multimodal speculative decoding that dynamically adapts the visual evidence provided to the draft model based on the current decoding state. This method achieved average speedups up to 1.68x over autoregressive decoding and improved average acceptance length across various MLLM backbones and multimodal benchmarks.
Key insights
- Visual demand in MLLMs is highly dynamic and non-monotonic, requiring an adaptive approach to visual evidence provision rather than fixed visual budgets.
- Dynamic selection of visual evidence based on the current decoding state significantly improves the quality and acceptance length of generated token sequences in speculative decoding.
- Effective integration of dynamic visual evidence is possible without disrupting the historical KV-cache, ensuring that the benefits of adaptive visual grounding are not offset by computational overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.