Reproduce “V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hangzhou He, Lei Zhu, Xinliang Zhang, Shuang Zeng, Qian Chen, Yanye Lu
The V2C-CBM framework constructs Concept Bottleneck Models by directly extracting concise, visually-oriented concepts from images using a Vision-to-Concept (V2C) tokenizer, bypassing verbose or non-visual issues of LLM-generated concepts. This method achieved 84.1% accuracy on ImageNet, outperforming existing CBMs and black-box linear probing, while providing inherent interpretability.
Key insights
- Generating concepts directly from vision using a V2C tokenizer eliminates the verbosity and non-visual attributes associated with LLM-derived concepts, enhancing interpretability.
- A carefully curated and filtered concept vocabulary, including bigrams and trigrams, significantly improves the model's ability to capture distinct visual attributes and classification performance.
- Leveraging unlabeled images with pre-trained Vision-Language Models (VLMs) enables scalable concept discovery without requiring extensive manual annotation or complex additional training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.