Reproduce “Continual Learning with Dependency Preserving Hypernetworks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dupati Srikar Chandra, Sakshi Varshney, P.K. Srijith, Sunil Gupta
A recurrent hypernetwork for continual learning generates target-model weights in dependent chunks rather than treating them as conditionally independent. The proposed LSTM-based variants, combined with Fisher-information-weighted regularisation or task-specific growth, consistently improve over chunked hypernetwork baselines across Split MNIST, Permuted MNIST, and CIFAR sequences, with LSTM NET GROW reaching 88.25% average accuracy on the reported CIFAR benchmark.
Key insights
- Modeling dependencies among generated weight chunks improves continual-learning performance, particularly when task identity is unavailable in domain- and class-incremental settings.
- Importance should be measured in the generated target network rather than applied uniformly to hypernetwork parameters, since target parameters differ in their contribution to task behavior.
- Network growth offers a speed–memory tradeoff: it avoids the increasing training cost of history-based regularisation but requires task-specific parameters whose storage grows with the task sequence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.