Reproduce “A Comprehensive Empirical Evaluation on Online Continual Learning” on OpenResearch

About this paper

Albin Soutif--Cormerais, Antonio Carta, Andrea Cossu, Julio Hurtado, Hamed Hemati, Vincenzo Lomonaco, Joost Van de Weijer

Researchers provided a comprehensive empirical evaluation of rehearsal-based Online Continual Learning (OCL) methods, introducing novel metrics for continual stability and representation quality. The study revealed that the performance gap in OCL primarily stems from challenges in the incrementally learned classifier, rather than a significant degradation of the learned latent representations.

Key insights

• The performance gap between OCL methods and i.i.d. training is largely attributable to issues with the incrementally learned classification layer, rather than a deterioration of the learned backbone features.
• OCL methods often exhibit characteristics of underfitting due to limited training iterations, rather than severe catastrophic forgetting, with evidence of backward transfer and stable cumulative forgetting.
• Implementation details, particularly the memory batch size, significantly influence OCL method performance, sometimes more than specific algorithmic modifications, highlighting the competitiveness of a well-tuned Experience Replay.

Summary by alphaXiv.