Reproduce “Fresh-CL: Feature Realignment through Experts on Hypersphere in Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhongyi Zhou, Yaxin Peng, Pin Yi, Minjie Zhu, Chaomin Shen
Fresh-CL introduces a framework that integrates Neural Collapse-inspired feature alignment with a Mixture of Experts to mitigate catastrophic forgetting and feature entanglement in continual learning. It achieves 87.0% average accuracy in full-shot Multi-task Incremental Learning, outperforming the previous state-of-the-art by 2.0%, and shows robustness in few-shot scenarios with a 1.1% gain.
Key insights
- Neural Collapse (NC) fosters optimal intra-domain feature separation by guiding class features and classifiers to converge to a Simplex Equiangular Tight Frame (ETF) structure.
- A single, fixed application of NC causes "projection degradation" in multi-domain continual learning, failing to maintain structured representations across diverse tasks.
- Integrating NC-inspired feature alignment with a Mixture of Experts (MoE) allows for adaptive, task-specific feature projections onto distinct hyperspherical subspaces, mitigating inter-domain interference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.