Reproduce “Zero-shot Concept Bottleneck Models” on OpenResearch

About this paper

Shin'ya Yamaguchi, Kosuke Nishida, Daiki Chijiwa, Yasutoshi Ida

Zero-shot Concept Bottleneck Models (Z-CBMs) enable human-interpretable and intervenable predictions by leveraging off-the-shelf vision-language models and a large concept bank, entirely bypassing target dataset training. These models achieved an average SigLIP-Score of 0.7754 for concept factuality and a 54.28% zero-shot classification accuracy across 12 diverse datasets, slightly exceeding standard zero-shot CLIP.

Key insights

• Interpretable and intervenable concept-based predictions can be achieved in a fully zero-shot manner by integrating frozen Vision-Language Models with a dynamically queried, large-scale concept bank.
• Sparse linear regression (Lasso) effectively selects a concise set of essential and mutually exclusive concepts from a large candidate pool, enhancing interpretability and concept factuality without explicit training.
• Approximating image features with a weighted sum of concept text features can reduce the modality gap inherent in VLMs, sometimes leading to improved classification performance even over direct VLM zero-shot classification.

Summary by alphaXiv.