Reproduce “A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michael V. DeBole, Rathinakumar Appuswamy, Neil McGlohon, Brian Taba, Steven K. Esser, Filipp Akopyan, John V. Arthur, Arnon Amir and 25 more
The authors present a rack-scale inference system that keeps model weights and key-value cache on NorthPole cards, passing intermediate tensors between cards. This design supports pipeline inference with small micro-batches. On Granite-3.3-8b-instruct, the system reports 2.8 ms inter-token latency at 2k context and batch size 28; its rack throughput and power claims are not matched comparisons with other systems.
Key insights
- The exclusive use of on-chip memory within the NorthPole accelerator is vital for achieving ultra-low-latency and energy-efficient inference by reducing costly off-chip data transfers.
- A co-designed hardware and software stack, from chip to cloud service, enables holistic optimization for performance, power efficiency, and practical data center deployability.
- Quantization-aware training, specifically the SiLQ algorithm, effectively mitigates accuracy degradation from aggressive quantization, allowing 4-bit weight models to match bfloat16 performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.