Reproduce “Mixture-of-Variational-Experts for Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heinke Hihn, Daniel A. Braun
The research introduces Hierarchical Variational Continual Learning (HVCL) with Mixture-of-Variational-Experts (MoVE) layers, enabling models to continuously acquire new knowledge while retaining old information, without requiring explicit task boundaries. This approach achieves competitive performance across supervised, generative, and reinforcement learning benchmarks in a task-agnostic setting.
Key insights
- A principled hierarchical information-theoretic framework can manage the balance between learning new information and preserving old knowledge without explicit task metadata.
- Variational experts within a sparsely-gated Mixture-of-Experts architecture can implicitly manage task separation and knowledge retention in a task-agnostic manner.
- Explicit diversity measures, particularly DPPs applied to expert parameter distributions, are effective in ensuring experts specialize in distinct areas, thereby reducing redundancy and enhancing knowledge retention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.