Reproduce “Parabolic Continual Learning” on OpenResearch

About this paper

Haoming Yang, Ali Hasan, Vahid Tarokh

Researchers at Duke University developed Parabolic Continual Learning (PCL), a framework that models a model's expected loss as a parabolic Partial Differential Equation (PDE) to address catastrophic forgetting. PCL achieved competitive accuracy with state-of-the-art methods while demonstrating superior robustness to noisy and imbalanced data, and remarkable efficiency with small memory buffers on various CIFAR and TinyImageNet benchmarks.

Key insights

• The expected loss profile of a continual learning model can be rigorously modeled as the solution to a parabolic Partial Differential Equation (PDE).
• Memory buffers in continual learning can serve as mathematically defined boundary conditions for the PDE, allowing for theoretical bounds on forgetting and generalization errors.
• The Feynman-Kac formula combined with Brownian bridges offers a computationally feasible way to optimize models under PDE constraints, effectively bridging regularization and replay methods.

Summary by alphaXiv.