Reproduce “Self-Supervised Models are Continual Learners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Enrico Fini, Victor G. Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, Julien Mairal

CaSSLe provides a framework for continual self-supervised learning, allowing models to learn visual representations from sequential, unlabeled data while effectively mitigating catastrophic forgetting. It integrates a distillation mechanism directly into existing self-supervised loss functions, yielding consistent performance improvements and broad compatibility with diverse state-of-the-art methods.

Key insights

  • Standard self-supervised learning objectives can be effectively adapted into distillation mechanisms for continual learning without introducing new loss-weighting hyperparameters.
  • A dedicated predictor network is essential to balance plasticity for new knowledge acquisition with stability for retaining previous representations.
  • The framework demonstrates broad compatibility across various SSL methods with different loss functions and collapse-avoidance strategies, highlighting its generalizability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.