Reproduce “NVLLM: A 3D NAND-Centric Architecture Enabling Edge on-Device LLM Inference” on OpenResearch

About this paper

Mingbo Hao, Changwei Yan, Haoyu Cui, Zhihao Yan, Yizhi Ding, Zhangrui Qian, Weiwei Shan

NVLLM performs feed-forward computation beside flash-stored weights, while attention and the key–value cache remain on a DRAM-side processor. This design targets repeated weight movement during single-token generation. In simulated INT8 decoding on LLaMA2-7B with a 64-token context, NVLLM-16C reaches 17.0 TPS versus AiF’s 13.1, a 1.3× advantage. This is a modeled result, not a prototype measurement.

Key insights

• Effective LLM inference on edge devices benefits from a heterogeneous memory hierarchy, strategically offloading FFNs to in-Flash computation and retaining attention mechanisms on DRAM-connected NPUs.
• It is feasible to perform numerically correct, mixed-precision computation directly on raw, error-prone 3D NAND data through an out-of-order, error-resilient dot-product engine that decouples error detection from correction.
• A clustered plane-lane co-design, optimized for data streaming and latency hiding, is crucial for maximizing internal 3D NAND bandwidth and amortizing CMOS logic overhead during FFN execution.

Summary by alphaXiv.