Reproduce “Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marcel Gröpl, Jaewoo Jung, Seungryong Kim, Marc Pollefeys, Sunghwan Hong
Researchers at ETH Zürich and KAIST AI developed Entropy-Gradient Grounding, a training-free method that enhances Vision-Language Models' ability to answer queries requiring fine-grained or multi-region visual evidence. This approach uses the VLM's predictive uncertainty to identify and iteratively refine relevant image regions, yielding substantial accuracy improvements of up to +20.81 on benchmarks like DocVQA and V* across various VLM architectures.
Key insights
- Backpropagating the VLM's predictive uncertainty (entropy) provides a robust, model-intrinsic signal for visual grounding, superior to heuristic attention-map interpretations.
- Iterative refinement guided by spatial entropy effectively addresses challenges of fine-grained and multi-region evidence, allowing the model to dynamically focus on relevant parts.
- The final layer's gradients are most effective for grounding, and backpropagating from the first predicted token yields the most robust relevance maps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.