Reproduce “K-LITE: Learning Transferable Visual Models with External Knowledge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sheng Shen, Chunyuan Li, Xiaowei Hu, Jianwei Yang, Yujia Xie, Pengchuan Zhang, Zhe Gan, Lijuan Wang and 6 more
K-LITE introduces a method to learn transferable visual models by augmenting natural language supervision with external knowledge from sources like WordNet and Wiktionary. This approach achieves absolute performance gains of 1-2% on ImageNet-1K and 2-6% on average across 20 downstream image classification datasets, while also demonstrating improved object detection performance with an average 2.8 AP improvement on LVIS.
Key insights
- External structured knowledge significantly improves the average concept overlap between pre-training and evaluation datasets, leading to better generalization for unseen concepts.
- Consistency between the use of knowledge during training and evaluation is crucial; a mismatch leads to significant performance degradation.
- The approach demonstrates sample efficiency, achieving comparable zero-shot performance with significantly fewer (e.g., half) pre-training image-text pairs, indicating that knowledge can compensate for less data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.