Reproduce “MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference” on OpenResearch

About this paper

Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, Mi Zhang

MEDA, from The Ohio State University's AIoT-MLSys-Lab, proposes a dynamic KV cache allocation method for Multimodal Large Language Models that leverages cross-modal attention entropy and a multimodal KV pair merging strategy. The approach reduces KV cache memory usage by up to 72% and achieves up to 2.82 times faster decoding speeds while maintaining or enhancing performance across multi-image and long-video tasks.

Key insights

• Cross-modal attention patterns vary significantly across different MLLM layers; early layers may have diffused attention, while deeper layers focus on specific, critical interactions.
• Dynamically allocating KV cache based on cross-modal attention entropy (inverse entropy softmax strategy) optimizes memory by adapting to each layer's unique attention distribution.
• Instead of discarding less important tokens, merging them (using average merging strategy) with conserved tokens preserves contextual information and mitigates performance degradation under high compression.

Summary by alphaXiv.