Reproduce “LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shichu Sun, Yichen Zhang, Haolin Song, Zonghao Guo, Chi Chen, Yidan Zhang, Yuan Yao, Zhiyuan Liu and 1 more
LLaVA-UHD v3 introduces a Progressive Visual Compression (PVC) framework for Multimodal Large Language Models, enabling efficient native-resolution image encoding that reduces inference latency by up to 2.4x while preserving or enhancing performance across various vision-language benchmarks.
Key insights
- Global native-resolution encoding (GNE) consistently outperforms slice-based encoding (SBE) in general semantic and especially spatial perception tasks, eliminating the 'cross-shaped bias' found in SBE for localization.
- The PVC framework effectively reconfigures pre-trained ViTs to support efficient native-resolution encoding, demonstrating that a balanced trade-off between fine-grained visual features and computational efficiency is achievable.
- The content-adaptive pooling within WTC is crucial for maintaining performance on fine-grained tasks during aggressive token reduction, improving expressiveness over simple average pooling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.