Reproduce “HOP to the Next Tasks and Domains for Continual Learning in NLP” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Umberto Michieli, Mete Ozay

The HOP method, developed by Samsung Research UK, enables NLP models to continuously learn from new tasks and domains while retaining previously acquired knowledge. It achieves this by employing high-order statistical moments of token embeddings and parameter-efficient adapter modules, demonstrating superior performance in maintaining accuracy and reducing catastrophic forgetting across various NLP applications and datasets compared to existing continual learning approaches.

Key insights

  • Capturing high-order statistical moments (like mean and variance) of token embeddings provides a richer and more robust representation for continual learning than relying solely on the [CLS] token.
  • Combining parameter-efficient adapters with problem-specific classification heads allows for effective continual adaptation across diverse NLP tasks and domains while mitigating catastrophic forgetting.
  • A unified approach can effectively manage both task-incremental and domain-incremental learning scenarios within the same framework, addressing a common limitation of prior methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.