Reproduce “Divide and not forget: Ensemble of selectively trained experts in Continual Learning” on OpenResearch

About this paper

Grzegorz Rypeść, Sebastian Cygert, Valeriya Khan, Tomasz Trzciński, Bartosz Zieliński, Bartłomiej Twardowski

This research from IDEAS-NCBR and affiliated universities introduces SEED (Selection of Experts for Ensemble Diversification), an exemplar-free continual learning method that employs a fixed-size ensemble of deep networks, selectively training one expert per task based on the separability of new class distributions. SEED achieves state-of-the-art performance in challenging class-incremental learning scenarios, particularly those with limited initial data or significant domain shifts, while maintaining computational efficiency and a strong balance between plasticity and stability.

Key insights

• Selectively training individual experts on new tasks, rather than updating all experts simultaneously, fosters greater diversification and specialization within the ensemble.
• Using multivariate Gaussian distributions to model class representations in latent space and employing a minimum overlap (KL-max) criterion for expert selection effectively promotes expert specialization and improves overall ensemble performance.
• A balance between plasticity and stability can be achieved in exemplar-free CIL by combining a frozen shared backbone with selectively updated, specialized expert heads and a knowledge distillation regularization term.

Summary by alphaXiv.