Reproduce “LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhongchun Zhou, Chengtao Lai, Wei Zhang

The LLaMCAT framework introduces cache arbitration and dynamic thread throttling to optimize Last-Level Cache (LLC) performance during Large Language Model (LLM) inference. This system specifically addresses Miss Status Handling Register (MSHR) contention and memory bandwidth limitations, achieving a geometric mean speedup of 1.26x over an unoptimized baseline in MSHR-bottlenecked scenarios and up to 1.58x in cache-size bottlenecked scenarios.

Key insights

  • MSHRs can be more effective than traditional cache storage in capturing temporal locality for LLM workloads, as MSHR hit lookup latency can overlap with DRAM access latency.
  • Dynamic thread throttling significantly enhances temporal locality of memory requests and improves the MSHR hit rate, reducing overall memory stalls by better managing request issuance.
  • The synergistic combination of balanced and MSHR-aware arbitration policies (BMA) with dynamic throttling (dynmg) provides greater performance benefits than individual policies, by effectively balancing fairness and MSHR utilization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.