Reproduce “Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hiroki Matsutani, Naoki Matsuda, Naoto Sugiura

Researchers at Keio University developed a distributed prompt caching mechanism to accelerate local Large Language Models (LLMs) on resource-constrained edge devices. This cooperative approach significantly reduced Time to First Token (TTFT) by 93.12% for a Gemma-3 270M model on a Raspberry Pi Zero 2W by leveraging cached intermediate states from a central server.

Key insights

  • Distributed prompt caching offers substantial performance gains for LLMs on highly resource-constrained edge devices, effectively offloading compute-intensive prompt decoding to cached states.
  • There is a critical trade-off between local processing power and communication overhead; for higher-end edge devices, downloading larger cache states via Wi-Fi can outweigh the benefits of bypassing local computation.
  • The Bloom-filter-based local catalog is essential for practical deployment, efficiently suppressing unnecessary network communication and making the distributed caching mechanism viable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.