Reproduce “MIND: Multi-Task Incremental Network Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jacopo Bonato, Francesco Pelosin, Luigi Sabetta, Alessandro Nicolosi
A replay-free parameter isolation method, MIND, mitigates catastrophic forgetting in deep neural networks by integrating distillation and task-specific batch normalization for each task's sub-network. Developed by Leonardo Labs researchers, it achieved superior accuracy across Class-Incremental and Domain-Incremental benchmarks, with one variant supporting memory-constrained devices.
Key insights
- Distillation plays a crucial role in effectively compressing and transferring knowledge from a teacher model into new task sub-networks, enhancing overall accuracy without requiring past data replay.
- Training task-specific Batch Normalization parameters for each sub-network is highly effective for robustly handling distributional shifts across different tasks.
- The combination of weight sharing among sub-networks and precise gradient guidance via a gating mechanism helps maintain a balance between network plasticity for new learning and stability for retaining past knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.