Reproduce “Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions” on OpenResearch

About this paper

Shuhao Gu, Bojie Hu, Yang Feng

A two-stage continual learning method was developed to update large pre-trained Neural Machine Translation models by identifying "low forgetting risk regions." This approach facilitates efficient adaptation to new domains or languages without accessing prior data or increasing model size, achieving a superior balance between new task performance and old knowledge preservation compared to regularization baselines.

Key insights

• Conventional regularization methods for continual learning often misestimate the true loss landscape, leading to suboptimal trade-offs in preserving old knowledge and acquiring new.
• Identifying parameter update regions based on local loss curvature (Fisher Information) or maintaining output stability (KL-divergence) enables more effective forgetting prevention.
• Hard-constraining parameter updates within these empirically determined "safe" regions is sufficient for large NMT models to adapt to new tasks without significant under-fitting or catastrophic forgetting.

Summary by alphaXiv.