Reproduce “Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves” on OpenResearch

About this paper

Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, Rio Yokota

Researchers from AIST and Tokyo Institute of Technology introduce "Visual Atoms," a novel methodology for pre-training Vision Transformers using systematically generated synthetic images. This approach creates a dataset that achieves 83.7% top-1 accuracy on ImageNet-1k fine-tuning with 21 million synthetic images, closely matching the 84.2% achieved by JFT-300M which uses 300 million real images.

Key insights

• Maximizing the diversity of contour shapes through varied frequency ranges ([0, 20]) and a broad range for the number of orbits ([1, 200]) significantly enhances the pre-training effectiveness of synthetic data for Vision Transformers.
• Adequate contour detail and smoothness, controlled by a higher quantization parameter ([200, 1000]), are beneficial for robust visual representation learning.
• Formula-Driven Supervised Learning can achieve pre-training effects comparable to large, real-world datasets while mitigating ethical, logistical, and accessibility concerns, providing a systematic way to improve synthetic dataset quality.

Summary by alphaXiv.