Reproduce “Stable Routing for Mixture-of-Experts in Class-Incremental Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zirui Guo, Quan Cheng, Da-Wei Zhou, Lijun Zhang
Nanjing University researchers introduce StaR-MoE, a framework addressing "routing drift" in expandable Mixture-of-Experts (MoE) models for class-incremental learning. It stabilizes routing for previously learned classes while effectively utilizing new expert capacity, achieving up to 4.57 average accuracy points improvement over prior methods on benchmarks like ImageNet-A.
Key insights
- Routing drift is a critical, often overlooked, structural cause of catastrophic forgetting in expandable MoE architectures for CIL, distinct from mere parameter updates.
- Stabilizing old-class routing (SARA) and effectively utilizing new expert capacity (ACR) are complementary and crucial for robust CIL in dynamic MoE systems.
- Adaptive weighting based on MoE layer sensitivity significantly improves the effectiveness of routing alignment constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.