Reproduce “HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng
HiDe, a training-free framework for Multimodal Large Language Models (MLLMs), addresses challenges in high-resolution image understanding by mitigating background interference and token redundancy through a hierarchical decoupling approach. The method enhances MLLM performance, boosting Qwen2.5-VL 7B's average accuracy on V*Bench by 13.0 percentage points, while significantly reducing peak GPU memory usage by 75% and halving inference latency compared to prior methods.
Key insights
- Cropping irrelevant background information and reducing redundant visual tokens are more critical for MLLM performance on high-resolution images than simply upscaling object sizes.
- Semantic query tokens provide more precise and localized visual grounding compared to aggregate attention, with mid-layers of the MLLM architecture yielding the strongest signals for region extraction.
- Preserving the relative spatial layout of multiple objects within a compact visual representation is crucial for accurate multi-object reasoning in MLLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.