Reproduce “Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models” on OpenResearch

About this paper

Md. Atabuzzaman, Andrew Zhang, Chris Thomas

Virginia Tech researchers improved zero-shot fine-grained image classification by introducing an iterative multiple-choice question-answering framework, attention intervention, and a novel class description curation for Large Vision-Language Models. This approach achieved state-of-the-art performance, outperforming previous methods by up to 29.64% absolute accuracy on specific datasets.

Key insights

• The iterative MCQA framework consistently outperformed direct class name generation and simple Yes/No QA, especially for open-source LVLMs, by structuring the decision-making process.
• Attention intervention can improve visual information flow and object-attribute grounding within LVLMs, though its effectiveness varies across different model architectures.
• High-quality, meticulously curated class descriptions are critical for achieving high performance in zero-shot fine-grained classification, demonstrating significant gains over existing descriptions.
• Proprietary LVLMs show strong performance with an "all-at-once" approach for many class descriptions, while open-source models benefit more from the iterative selection due to context window and reasoning limitations.

Summary by alphaXiv.