Reproduce “Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Uranik Berisha, Jens Mehnert, Alexandru Paul Condurache
Leveraging activation clustering and variance-based expert extraction, a data-driven method converts pretrained Vision Transformers into efficient Mixture-of-Expert models after initial training. This approach achieves MACs reductions up to 36% and parameter reductions up to 32% on DeiT-B, recovering over 98% of original accuracy with minimal fine-tuning.
Key insights
- Pretrained Vision Transformers exhibit emergent modularity in their deeper Multi-Layer Perceptron (MLP) layers, allowing for effective post-training expert extraction.
- A data-driven expert configuration, utilizing HDBSCAN and variance-based selection, achieves more balanced expert utilization compared to prior "MoEfication" methods.
- The methodology effectively generalizes to various ViT architectures (Swin, ConvNeXt) and datasets (CIFAR-100), demonstrating broad applicability for efficient model adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.