Reproduce “FORGE: Procedural Weight Generation for Sparse MoE Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chanyoung Moon, Travis Pence, Daisuke Yamada, Jiaqi Mo, Yanxin Zhang, Vikas Singh, Karthikeyan Sankaralingam
FORGE replaces some mixture-of-experts feed-forward weights with compact rotor recipes, regenerating approximate tiles on-chip with a dedicated Rotor Generation Unit. It targets short-context single-token decoding, where sparse routing can leave Tensor Cores underused as weight transfers dominate. Across five models, modeled throughput rises 1.3–2.8× at matched GPU count with 30% replacement, but tested quality is worse than rank-8 LoRA-only.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.