Reproduce “RoMe: Row Granularity Access Memory System for Large Language Models” on OpenResearch

About this paper

Hwayong Nam, Seungmin Baek, Jumin Kim, Michael Jaemin Kim, Jung Ho Ahn

RoMe is an HBM system for LLM serving that transfers whole DRAM rows instead of 32-byte lines. It simplifies controller scheduling and uses freed command/address pins to add channels, raising bandwidth from 2 TB/s to 2.25 TB/s. Simulations report lower decode time per output token for three models, including 10.4% for DeepSeek-V3; prefill changes by at most 0.1%, and the decode gains largely reflect added bandwidth.

Key insights

• LLM workloads exhibit predominantly coarse-grained, sequential memory access patterns that can be efficiently served by a row-level memory granularity, rather than traditional fine-grained designs.
• Simplifying the memory interface and access granularity can drastically reduce the complexity, area, and power consumption of the memory controller.
• Repurposing physical pins, freed by interface simplification, offers a viable and cost-effective method to scale aggregate memory bandwidth for domain-specific applications like LLMs.

Summary by alphaXiv.