Reproduce “Continual Learning with Invertible Generative Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jary Pomponi, Simone Scardapane, Aurelio Uncini
A continual learning method called Pseudo-Rehearsal Embedding Regularization (PRER) combines generative pseudo-rehearsal and regularization by training Normalizing Flows on the internal embedding space of a neural network. This approach, developed by researchers at Sapienza University of Rome, achieves performance comparable to or better than state-of-the-art methods on standard CL datasets while demonstrating constant and significantly reduced memory overhead.
Key insights
- Training a generative model (Normalizing Flow) in the network's lower-dimensional embedding space is more stable and effective than in the high-dimensional input space.
- Regularizing the current network's internal embeddings using synthetic embeddings generated from past tasks effectively mitigates catastrophic forgetting.
- Conditioning the decoder component of the generative model on class labels, rather than the Normalizing Flow itself, yields optimal performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.