Reproduce “Pre-training Vision Transformers with Formula-driven Supervised Learning” on OpenResearch

About this paper

Hirokatsu Kataoka, Sora Takashima, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Xinyu Zhang, Edgar Josafat Martinez-Noriega, Nakamasa Inoue and 1 more

Researchers at the National Institute of Advanced Industrial Science and Technology (AIST) and Institute of Science Tokyo developed a Formula-driven Supervised Learning (FDSL) method to pre-train Vision Transformers entirely on synthetic data. This approach achieved 82.7% top-1 accuracy on ImageNet-1k with ViT-Base models, outperforming ImageNet-21k pre-training, and neared JFT-300M performance with significantly fewer images.

Key insights

• ViTs learn effective visual representations from synthetic data primarily by focusing on clear object contours, with attention maps showing emphasis on these shapes and performance significantly degrading with broken or overly simple/complex contours.
• Increased parameter complexity and diversity in formula-driven data generation, achieved through more intricate mathematical formulas and random viewpoint projections, consistently improve pre-training effectiveness.
• FDSL offers a viable, ethical, accessible, and resource-efficient pre-training paradigm, providing a robust alternative to conventional supervised and self-supervised learning methods that depend on real-world images.

Summary by alphaXiv.