Reproduce “Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Oanh N. Tran, Thanh Quoc Hung Le, Oscar Chew, Kuan-Hao Huang, Khoa D. Doan
ViRGo, a lightweight framework, dynamically routes Vision-Language Model (VLM) inputs to appropriate perception strategies, like global processing or selective visual retrieval. This system boosts VLM accuracy by up to 8.6% on VQA benchmarks, particularly for small objects, while cutting inference time by as much as 49.9% compared to always-on retrieval methods.
Key insights
- A systematic analysis revealed that different VLM inference paradigms exhibit distinct scale-dependent failure modes, confirming an inherent "resolution–context trade-off" in visual reasoning.
- Intrinsic, zero-shot signals from a VLM's initial forward pass, such as implicit object scale and semantic token confidence, can reliably inform dynamic routing decisions without requiring additional training.
- Adaptive routing, guided by these dynamically extracted signals, enables VLMs to achieve a superior accuracy-efficiency balance by selectively applying computationally intensive retrieval methods only when necessary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.