Reproduce “Rethinking Momentum Knowledge Distillation in Online Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicolas Michel, Maorong Wang, Ling Xiao, Toshihiko Yamasaki
The authors propose MKD for online continual learning, adding a teacher whose weights track the learner through an exponential moving average rather than saving task-end snapshots. The teacher’s predictions regularize the learner, and both models’ weights are averaged for inference. On ImageNet100 with memory 5,000, MKD raises GSA’s accuracy by 11.52 points, but exceeds GSA+SDP by 1.14 points.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.