Reproduce “ToolFG: Towards Well-Grounded Fine-Grained Image Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Xue, Haoxuan Qu, Zhuoling Li, Yihang Lou, Yan Bai, Hossein Rahmani, Jun Liu
A framework called ToolFG empowers Multimodal Large Language Models (MLLMs) to perform fine-grained image classification by autonomously using external tools to gather verifiable visual evidence. This approach consistently surpasses state-of-the-art methods across five benchmarks, achieving an average harmonic mean of 87.9% in zero-shot per-dataset training, an 8.1 percentage point improvement over the previous best.
Key insights
- Equipping MLLMs with external, quantifiable tools enables a transition from vague qualitative reasoning to collecting verifiable, objective evidence for fine-grained classification.
- A novel MCTS-guided knowledge distillation method, combined with trajectory-level contrastive learning, can effectively transfer complex tool-use behaviors from advanced MLLMs to smaller models.
- Iterative co-evolution of the MLLM's tool-use policy and the toolset, guided by usage feedback and groundedness rewards, leads to robust and optimized fine-grained classification capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.