Reproduce “The Effect of Task Ordering in Continual Learning” on OpenResearch

About this paper

Samuel J. Bell, Neil D. Lawrence

Researchers at the University of Cambridge investigated how the order of tasks influences catastrophic forgetting in continual learning, demonstrating that an "anti-curriculum" strategy, where tasks are ordered to maximize the accumulated curvature between successive task minima, consistently reduces forgetting across various datasets by up to 10.5% compared to ordering by minimal curvature.

Key insights

• Task ordering is a critical, often-neglected hyperparameter in continual learning, significantly impacting performance and explaining more variance than random initialization seeds.
• An "anti-curriculum" approach, which orders tasks to maximize dissimilarity or "disruption" as measured by accumulated curvature, is more effective at reducing catastrophic forgetting than a traditional curriculum.
• Deliberately choosing task transitions that result in higher curvature between successive task minima helps maintain distinct representations, thereby mitigating the representational overlap that drives forgetting.

Summary by alphaXiv.