Reproduce “Continual Learning in the Teacher-Student Setup: Impact of Task Similarity” on OpenResearch

About this paper

Sebastian Lee, Sebastian Goldt, Andrew Saxe

The authors model catastrophic forgetting in sequential learning with teacher–student networks, controlling overlap between tasks’ feature weights and readouts. They derive learning dynamics in a large-input limit and numerically study feature and readout similarity. In these synthetic two-task experiments, intermediate similarity yields more forgetting, but the pattern depends on similarity type, measurement time, and training setup.

Key insights

• The counter-intuitive empirical finding that intermediate task similarity causes maximal catastrophic forgetting is reproduced and mechanistically explained within the theoretical model.
• Feature similarity and readout similarity have distinct and complex impacts on forgetting and transfer, with their relative balance influencing the severity of forgetting for a given total similarity.
• Readout weights learned for a prior task can bias the student's feature representation, influencing the learning of subsequent tasks even when a new readout head is used.

Summary by alphaXiv.