Reproduce “Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cong Li, Yihan Yin, Chenhao Xue, Zhao Wang, Fujun Bai, Yixin Guo, Xiping Jiang, Qiang Wu and 2 more
HeLioS, a hybrid-bonding-based LLM serving accelerator, integrates a 3D DRAM stack with a customized logic die and Processing Engines to efficiently handle dynamic LLM serving workloads. The system achieves a 3.25x geomean speedup and 3.36x better energy efficiency compared to leading GPU and NMP baselines by employing NMP-native dynamic KV cache management and fine-grained tiled attention.
Key insights
- Hybrid bonding technology provides superior bandwidth and customizability, making it a viable foundation for specialized LLM accelerators that can overcome the memory and compute limitations of traditional NMP designs.
- Overcoming the fundamental mismatch between NMP architectures and dynamic LLM serving requires a co-designed approach where KV cache management and attention computation are natively optimized for distributed memory and processing elements.
- Spatially-aware KV cache allocation, combined with fine-grained tiled attention, significantly reduces memory fragmentation and improves compute load balance across PEs, which is crucial for efficient LLM serving under dynamic workloads.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.