Reproduce “Potential Applications of HBF in LLM Serving Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yihan Yin, Yinlun Zhao, Zhixin Yun, Guanying Wu, Feng Zhu, Kai Tao, Shu Li, Fei Huang and 3 more
The research explores integrating High-Bandwidth Flash (HBF) into GPU memory hierarchies to augment capacity beyond HBM for Large Language Model (LLM) serving. It demonstrates that HBF-backed memory can reduce MoE decoding latency by 10.5% and dramatically decrease Time-To-First-Token by 28.7x in multi-model serving by enabling greater model and expert residency.
Key insights
- An 'integrated HBM/HBF stack' is identified as the most promising HBF integration strategy, ensuring HBF's capacity advantage is realized without degrading HBM-critical performance.
- Increased resident expert capacity, enabled by HBF, improves MoE decoding latency by enhancing expert access locality and reducing cross-node communication overheads.
- Expanded capacity for multi-model serving drastically reduces model loading/eviction latency and allows for proactive weight replication, improving request load balancing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.