Reproduce “AI and Memory Wall” on OpenResearch

About this paper

Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer

Research from UC Berkeley demonstrates that the historical 'memory wall' is re-emerging as the dominant bottleneck for AI, especially in Large Language Model (LLM) serving, due to the widening gap between rapidly scaling computational power and much slower memory bandwidth improvements. The work quantifies this disparity and empirically shows how specific Transformer workloads become memory-bound in practice.

Key insights

• The memory wall has resurfaced as the dominant performance bottleneck for many AI applications, particularly for serving Large Language Models (LLMs) at low batch sizes.
• Simply increasing raw computational power (FLOPS) without commensurate improvements in memory and communication infrastructure is an unsustainable path for AI innovation.
• Arithmetic intensity (FLOPs per byte loaded from memory) is a critical metric for understanding hardware utilization, revealing that many AI workloads are memory-bound even with high FLOPs.

Summary by alphaXiv.