Reproduce “ExFusion: Efficient Transformer Training via Multi-Experts Fusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiacheng Ruan, Daize Dong, Xiaoye Qu, Tong Zhu, Ting Liu, Yuzhuo Fu, Yu Cheng, Suncheng Xiang
ExFusion trains image-classification Transformers by blending the parameters of multiple expert feed-forward networks into one network for each forward pass, then retaining only the fused network for inference. On ImageNet-1k, ExFusion-mb ViT-B/16 reaches 81.1% Top-1 accuracy versus 77.9% for dense ViT-B/16, with 264 versus 240 GPU-hours and the same 86.6M parameters. These results concern the reported ImageNet setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.