Reproduce “Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy” on OpenResearch

About this paper

Danni Peng, Kai Wu, Tianyu Zuo, Pengfei Xia, Hui Zang

The authors present admission-controlled LRU-K for generative recommendation, which reuses users’ key-value caches while rejecting some missed caches unless request history indicates reuse. Rejected misses still recompute history but avoid full-cache writes. In an analytical model, LRU-10 extends estimated HBF lifetime from about one to over six years, with comparable modeled throughput, under assumed workloads and hardware.

Key insights

• HBF’s capacity and 1.6 TB/s modeled read bandwidth allow it to retain many more user histories than HBM, whereas a CPU tier can become slower than recomputation when accessed through a 128 GB/s link.
• Cache misses should not automatically trigger durable writes: separating miss-with-write from miss-without-write avoids spending flash endurance on low-reuse users.
• Requiring repeated accesses can improve cache value as well as endurance because frequently requested users tend to have longer histories, making their retained KV states more costly to recompute.

Summary by alphaXiv.