Reproduce “Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yifan Wang, Peiming Li, Shiyu Li, Zhiyuan Hu, Xiaochen Yang, Wenming Yang, Yang Tang, Zheng Wei

Researchers from Tencent BAC and affiliated universities developed LazyMCoT, a dynamic and training-free framework for visual grounding in Multimodal Large Language Models (MLLMs). This framework adaptively routes visual queries to either a direct MLLM response or a collaborative grounding module, improving overall accuracy by 3.3 to 8.4 points across various MLLMs while reducing inference latency.

Key insights

  • The MLLM's predictive uncertainty, quantifiable via "zero-cost" features from the first answer token's logits, serves as an effective and efficient indicator for adaptively routing visual grounding efforts.
  • Selectively applying complex visual grounding operations prevents performance degradation on simpler queries that benefit from the global context of the base MLLM.
  • A synergistic approach combining the MLLM's inherent cross-modal attention with an external visual expert and a two-stage refinement strategy can create more accurate and comprehensive localized visual evidence for fine-grained perception.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.