Reproduce “Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shubham Agarwal, Sai Sundaresan, Subrata Mitra, Debabrata Mahapatra, Archit Gupta, Rounak Sharma, Nirmal Joshua Kapu, Tong Yu and 1 more

Cache-Craft introduces a chunk-caching system to improve RAG inference efficiency by optimizing the prefill phase, the primary bottleneck in RAG workloads. Developed by researchers at Adobe Research, IIT Bombay, and IIT Kanpur, the system maintains 90-95% output quality while reducing Time-to-First-Token latency by up to 2-3x and increasing throughput by up to 1.6x.

Key insights

  • RAG systems are heavily prefill-bound, with prefill processing consuming up to 90% of total inference time, making it the primary bottleneck.
  • Traditional prefix caching is ineffective in RAG due to the dynamic and varied nature of retrieved chunks, leading to rare exact prefix matches.
  • Attention patterns vary across transformer layers, with middle layers showing stronger chunk-specific attention, guiding where and how chunk caches can be reused or recomputed.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.