Reproduce “Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou

Researchers at DAMO Academy, Alibaba Group, developed Chinese CLIP, a vision-language foundation model tailored for the Chinese language that significantly outperforms previous Chinese multimodal baselines on cross-modal retrieval benchmarks, achieving gains of up to 17.0 Mean Recall on MUGE. The model also demonstrates competitive zero-shot image classification capabilities across 20 diverse datasets, sometimes surpassing English foundation models.

Key insights

  • Leveraging pretrained weights for model initialization is critical for efficient pretraining and achieving superior performance, significantly outperforming training from scratch.
  • The two-stage pretraining method, particularly the full 'Contrastive Tuning' stage where both encoders are optimized, is essential for effectively adapting the vision encoder to the distinct distribution of Chinese multimodal data.
  • Scaling model size and input image resolution consistently improves performance across both cross-modal retrieval and zero-shot image classification tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.