Reproduce “Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Xiaodan Liang, Lewei Yao, Runhui Huang and 4 more
Researchers from Huawei Noah's Ark Lab and Sun Yat-sen University created Wukong, the first publicly available 100-million-scale Chinese cross-modal dataset. They also developed dual-stream Vision-Language Pre-training (VLP) models trained on Wukong, demonstrating strong zero-shot classification and image-text retrieval performance, often outperforming existing Chinese multimodal baselines.
Key insights
- Character-grained tokenization for Chinese text proved more effective than word-grained tokenization, providing finer-grained semantic representations for VLP models.
- Locked-image Text tuning (LiT-tuning) consistently led to higher validation accuracy, demonstrating effective generalization by adapting powerful English-pretrained image encoders to Chinese text.
- The models learned meaningful fine-grained alignments between Chinese text tokens and image patches, as visualized through token-wise similarity, and showed some alignment to English tokens due to multilingual vocabulary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.