Reproduce “Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hannah Atmer, Yuan Yao, Thiemo Voigt, Stefanos Kaxiras
Research from Uppsala University quantifies the energy efficiency and performance trade-offs of on-chip SRAM size and operating frequency for Large Language Model inference's prefill and decode phases. It reveals that decode performance is heavily capped by memory bandwidth, while larger SRAMs disproportionately increase static energy, leading to an optimal configuration of 32-64KB local SRAM and 1200-1400MHz for minimal Energy-Delay Product under baseline conditions.
Key insights
- LLM prefill is compute-bound, benefiting from higher frequencies, while decode is memory-bound, with performance plateauing beyond 400-600 MHz due to external memory bandwidth limits.
- On-chip SRAM size is the primary driver of total energy consumption, with larger SRAMs leading to significantly increased static (leakage) energy.
- For compute-bound phases, increased frequency can counter-intuitively reduce total energy by decreasing execution time and thus static energy.
- Memory bandwidth acts as a critical ceiling for decode performance, dictating the maximum effective compute frequency and influencing overall efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.