Reproduce “Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints” on OpenResearch

About this paper

Sunil Kumar, Bowen Zhao, Leo Dirac, Paulina Varshavskaya

Researchers at Groundlight AI trained a smaller Vision-Language Model to use a "zoom-in" tool for enhanced detailed visual reasoning under computational resource constraints. This approach achieved 80.1% accuracy on the V*Bench dataset, representing a 5.7% improvement over the baseline and demonstrating parameter efficiency comparable to larger models.

Key insights

• Reinforcement learning can effectively teach smaller VLMs to actively use external tools for detailed visual reasoning, circumventing the need for pre-annotated tool-use trajectories.
• A simplified, YAML-like tool-calling interface with keypoint input significantly improves tool invocation stability and reduces formatting errors compared to complex JSON-based bounding box inputs.
• Strategic training data selection, specifically over-representing visually challenging examples combined with a structured reward function, is critical for successful tool-use learning and improved generalization.

Summary by alphaXiv.