Reproduce “Few-Shot Image Classification and Segmentation as Visual Question Answering Using Vision-Language Models” on OpenResearch

About this paper

Tian Meng, Yang Tao, Ruilin Lyu, Wuliang Yin

The VISE method, developed by researchers from The University of Manchester and Mettler Toledo Safeline, re-conceptualizes few-shot image classification and segmentation as a visual question answering problem, employing a training-free orchestration of Vision-Language Models with off-the-shelf vision tools. This approach yielded state-of-the-art results on Pascal-5i and COCO-20i datasets, significantly improving segmentation mIoU by up to 24.7% and 26.4% respectively.

Key insights

• Few-shot image classification and segmentation can be effectively solved as a training-free Visual Question Answering problem, shifting away from computationally intensive training-dependent methods.
• Vision-Language Models can serve as high-level orchestrators and reasoners for complex visual tasks by compositionally integrating with specialized, high-performance off-the-shelf vision tools.
• The synergistic combination of VLM reasoning and dedicated vision tools (like YOLO for detection and SAM for segmentation) is crucial for achieving high accuracy in few-shot scenarios, significantly outperforming VLMs used in isolation for generating visual primitives.

Summary by alphaXiv.