Reproduce “Pre-training without Natural Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, Yutaka Satoh
This research introduces Formula-driven Supervised Learning, a method that pre-trains convolutional neural networks using automatically generated fractal images and their mathematical labels, completely bypassing natural image data. Models pre-trained on this synthetic FractalDB achieve performance competitive with, and in some cases superior to, models pre-trained on established natural image datasets for various downstream tasks, while inherently addressing privacy, copyright, and ethical concerns.
Key insights
- Convolutional Neural Networks can learn effective visual representations from abstract, mathematically generated patterns, challenging the long-held assumption that natural images are essential for pre-training.
- Formula-driven supervised learning offers a scalable, cost-effective, and ethically sound alternative to natural image datasets by inherently avoiding privacy, copyright, and bias concerns.
- Models pre-trained on fractal-generated data learn distinct low-level features yet still generalize well to natural image tasks, implying diverse pathways for acquiring useful visual representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.