Reproduce “Expandable and Differentiable Dual Memories with Orthogonal Regularization for Exemplar-free Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyung-Jun Moon, Sung-Bae Cho
An exemplar-free continual learning method, Expandable and Differentiable Dual Memories (EDD) with Orthogonal Regularization, was developed to mitigate catastrophic forgetting and promote knowledge transfer. This approach consistently surpassed 14 baseline methods on standard benchmarks like CIFAR-100 and TinyImageNet, improving final accuracy on S-CIFAR-100 (10-task) by 6.16 percentage points compared to the closest competitor.
Key insights
- Decomposing data features into smaller sub-features and storing them in differentiable memories facilitates the extraction of diverse patterns, promoting knowledge reuse across tasks.
- Enforcing geometric separation through orthogonal regularization between preserved and newly learned memory components effectively prevents representational interference.
- An adaptive memory adjustment module, which prunes critical slots and minimally expands capacity, balances stability and plasticity while preventing unbounded model growth.
- Memory-guided representation alignment, by aligning internal memory activation patterns, consolidates knowledge across tasks without requiring exemplars.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.