Reproduce “DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyang Lu, Belthangady Akash Vi Narayana Pai, Xian-He Sun
DynaNDE serves mixture-of-experts language models by sending activations to a near-data processor for expert computation while a neural processing unit handles other experts. Its per-layer scheduler models computation, transfers, overlap, and cached expert reuse to choose the split. Across three evaluated models, it reports 2.6× prefill and 2.2× decoding throughput over MoNDE in cycle-accurate simulation.
Key insights
- A quantitative analytical model capturing NPU-NDP hardware heterogeneity and communication-computation overlap enables more informed, adaptive expert scheduling than fixed heuristic-based approaches.
- Dynamically assigning experts to compute units based on their expected performance gain (preference score) and awareness of expert reuse in NPU memory significantly reduces redundant data movement and improves throughput.
- The structured three-phase co-execution model for NPU-NDP interaction effectively manages contention on shared interconnects, ensuring accurate latency prediction and efficient resource utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.