Reproduce “Improving and Understanding Variational Continual Learning” on OpenResearch

About this paper

Siddharth Swaroop, Cuong V. Nguyen, Thang D. Bui, Richard E. Turner

VCL carries a distribution over weights forward between tasks, using each task’s approximate posterior as the next task’s prior. Its revised training recipe combines longer optimization, local reparameterization, and small random weight initialization. On 10-task permuted MNIST, final average test accuracy across tasks is 93±1% over five runs, versus 90% previously; many hidden units appear inactive in these MNIST experiments.

Key insights

• VCL exhibits an emergent pruning effect, where hidden units are made inactive (their weights revert to the prior) for specific tasks, demonstrating efficient and adaptive model capacity allocation.
• The variational objective inherently encourages sparse solutions, suggesting that pruning is an optimal strategy within the VCL framework for minimizing KL divergence and managing capacity.
• Current continual learning benchmarks like Permuted MNIST are highly sensitive to model capacity, and excessive capacity can mask critical challenges, necessitating more constrained benchmark settings for meaningful evaluation.

Summary by alphaXiv.