Reproduce “CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecture” on OpenResearch

About this paper

András Kalapos, Bálint Gyires-Tóth

CNN-JEPA introduces an adaptation of the Joint Embedding Predictive Architecture for Convolutional Neural Networks, learning representations by predicting latent features of masked regions. This method achieved competitive accuracy on ImageNet-100 and ImageNet-1k while reducing training time by 17-35% and operating with a substantially smaller model footprint compared to other CNN self-supervised learning approaches.

Key insights

• The JEPA framework can be effectively adapted to CNNs by integrating sparse convolutions for masked inputs and designing parameter-efficient depthwise separable predictors for latent feature prediction.
• Learning patch-level latent features through prediction offers a balanced approach to feature representation, combining aspects of both fine-grained detail and broader semantic understanding.
• Simplifying the pretraining architecture, reducing reliance on extensive data augmentations, and omitting a separate projector network can yield substantial computational efficiency gains for CNN-based SSL without sacrificing competitive performance.

Summary by alphaXiv.