Reproduce “On the Computation of the Fisher Information in Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gido M. van de Ven
This analysis details how different methods for computing the Fisher Information (FI) matrix impact the performance of Elastic Weight Consolidation (EWC) in continual learning. It finds that exact FI computation significantly improves EWC's effectiveness on complex tasks compared to commonly used empirical or batched approximations.
Key insights
- The method used for Fisher Information computation substantially impacts EWC's performance, particularly on more challenging continual learning tasks like Split CIFAR-10, where exact methods yield superior accuracy.
- For practical use, exactly computing the Fisher Information on a subset of data (e.g., EXACT (n=500)) provides an effective balance, achieving near-optimal accuracy with reduced computational cost.
- Different Fisher Information computation strategies require vastly different optimal regularization strengths (hyperparameter λ), underscoring the importance of transparent implementation details and careful hyperparameter tuning for reproducible research.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.