Reproduce “S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Giang Do, Hung Le, Truyen Tran
Researchers from Deakin University developed S2MoE, a method for Sparse Mixture of Experts that injects stochasticity into input features to mitigate representation collapse and enhance expert learning diversity. S2MoE consistently outperforms existing SMoE baselines across NLP tasks, achieving competitive performance with a 28% reduction in inference FLOPs.
Key insights
- Injecting stochasticity directly into input representations can effectively diversify the features presented to experts, fostering more distinct learning than solely improving router mechanisms.
- A dual learning pathway, which processes both original and noise-augmented inputs, allows for a learned balance between stable feature learning and the exploration of varied input signals.
- By considering both original and noise-augmented inputs, the method effectively expands the representational space available to experts, which theoretically and empirically mitigates representation collapse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.