Reproduce “Overcoming catastrophic forgetting with hard attention to the task” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joan Serrà, Dídac Surís, Marius Miron, Alexandros Karatzoglou

Researchers at Telefónica Research developed Hard Attention to the Task (HAT), a mechanism that equips neural networks with an attention-based approach to mitigate catastrophic forgetting during sequential task learning. This method learns to dynamically activate specific network units for each task, leading to significantly reduced forgetting rates and providing insights into network capacity utilization.

Key insights

  • Learning task-specific attention at the unit level provides a flexible and granular way to create and preserve computational paths for different tasks.
  • Modifying gradients based on cumulative attention from past tasks effectively prevents catastrophic forgetting by safeguarding critical weights.
  • The use of annealed sigmoid functions and a regularization term allows for learning distinct, sparse, and reusable attention masks concurrently with network weights.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.