Reproduce “Seeing in Words: Learning to Classify through Language Bottlenecks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khalid Saifullah, Yuxin Wen, Jonas Geiping, Micah Goldblum, Tom Goldstein
Researchers from the University of Maryland and New York University developed a novel image classification approach that processes visual features through a 'language bottleneck,' generating discrete word sequences that serve as intermediate representations for classification. This method, trained end-to-end with class-level supervision, achieved 68.894% accuracy on ImageNet and demonstrated improved robustness on ImageNet-C, outperforming a direct caption-based classification baseline.
Key insights
- It is feasible to train an image classification model to use human-readable text as an intermediate feature representation, enabling intrinsic interpretability of its decision-making process.
- Optimizing for diversity among generated tokens, using a 'Token Similarity' loss, improves both classification accuracy and the interpretability of the word sequences.
- A trade-off exists between generating linguistically fluent (human-readable) text and maximizing the discriminative power of the words for classification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.