Reproduce “TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qinying Liu, Wei Wu, Kecheng Zheng, Zhan Tong, Jiawei Liu, Yu Liu, Wei Chen, Zilei Wang and 1 more
TagAlign is a framework that improves vision-language (V&L) model alignment by employing LLM-aided parsing to extract fine-grained object and attribute tags from image captions. This approach demonstrated enhanced zero-shot semantic segmentation by an average of 6.3% mIoU on background-inclusive datasets and improved referring segmentation, especially for attribute-specified objects, without requiring manual annotations.
Key insights
- LLM-aided parsing provides higher quality and more comprehensive object and attribute tags from raw captions compared to traditional NLTK-based methods, significantly enhancing supervision signal quality.
- Directly supervising V&L models with diverse and automatically generated multi-tag classifications, particularly distinguishing between objects and attributes, is critical for achieving fine-grained semantic localization.
- Leveraging CLIP's zero-shot text encoder capabilities to generate fixed tag embeddings for the classifier efficiently aligns visual features with a broad, open-vocabulary set of object and attribute concepts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.