Reproduce “LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics” on OpenResearch

About this paper

Randall Balestriero, Yann LeCun

LeJEPA learns representations by aligning related views while regularizing their population toward an isotropic Gaussian, using SIGReg to compare random one-dimensional projections and a single trade-off parameter. The authors’ theory supports this geometry for specified linear and nonlinear probe families, not all tasks. On Galaxy10, frozen-backbone linear accuracy is 78.17% versus DINOv3’s 71.38%, under unmatched model and pretraining conditions.

Key insights

• The isotropic Gaussian distribution is theoretically optimal for encoder embeddings, minimizing integrated squared bias and variance for various downstream evaluation tasks.
• High-dimensional distribution matching can be achieved scalably and stably by testing one-dimensional projections of embeddings (via a modified Cramér-Wold theorem), particularly with the Epps-Pulley test which ensures uniformly bounded gradients.
• A principled, theoretically-grounded distribution regularization strategy effectively prevents representation collapse and achieves strong performance without requiring the complex, brittle heuristics common in other SSL methods.

Summary by alphaXiv.