Reproduce “Learning Vision from Models Rivals Learning Vision from Data” on OpenResearch

About this paper

Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, Phillip Isola

Researchers from Google Research and MIT CSAIL developed SynCLR, a method for learning general-purpose visual representations entirely from synthetic images and captions generated by other AI models. The approach achieved competitive performance on benchmarks like ImageNet linear evaluation and fine-grained classification, demonstrating a viable alternative to reliance on real-world data.

Key insights

• High-quality, general-purpose visual representations can be learned exclusively from purely synthetic data, challenging the necessity of real-world data for state-of-the-art performance.
• Careful synthesis of captions using LLMs and prompt engineering is crucial for generating effective training data for visual representation learning, outperforming naive synthetic or real captions.
• Defining 'visual classes' with caption-level granularity, which captures more nuanced semantics, significantly outperforms instance-level or coarse-grained class definitions for learning robust representations.

Summary by alphaXiv.