Reproduce “Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhui Lv, Yunming Liao, Hongli Xu, Yang Xu, Junhao Cheng
Apex improves MoE inference when GPU memory cannot hold all experts by assigning missing experts between CPU execution and GPU execution. It uses each expert’s routed-token load to balance CPU computation against GPU weight transfers and computation, while speculative prefetching anticipates next-layer demand. Across three BF16 models and batch sizes, it improves prefill speed by 31.2% and decode throughput by 31.1% over the strongest baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.