Reproduce “ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jewon Lee, Wooksu Shin, Seungmin Yang, Ki-Ung Song, DongUk Lim, Jaeyeon Kim, Tae-Ho Kim, Bo-Kyeong Kim
Researchers from Nota Inc. developed ERGO, a framework enhancing high-resolution visual understanding in Large Vision-Language Models (LVLMs) by employing a reasoning-driven perception reward within a two-stage coarse-to-fine pipeline. This approach enabled ERGO to achieve superior accuracy on high-resolution benchmarks with fewer vision tokens and up to 3x faster inference times compared to existing methods.
Key insights
- Multimodal contextual information is crucial for accurately identifying task-relevant regions in downsampled images, particularly when direct visual cues are ambiguous or absent.
- A carefully designed reinforcement learning reward function, specifically the Task-driven Contextual Exploration (TCE) reward, effectively guides LVLMs to perform efficient and accurate 'reasoning-driven perception' by focusing on informative regions.
- Prioritizing the quality and efficiency of the selected high-resolution region through dedicated rewards significantly improves overall performance in a coarse-to-fine pipeline, complementing end-to-end task accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.