Reproduce “ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shao Wang, Rui Ren, Lin Gui
ForkKV introduces a memory management framework for multi-LoRA agent serving that disaggregates the Key-Value (KV) cache into shared base and agent-specific residual components. This approach reduces per-agent memory footprint by 12.7x, leading to up to 3.0x higher throughput for multi-agent workflows with a negligible 0.71 F1-score drop in generation quality.
Key insights
- Algebraic decomposition of LoRA projections reveals a massive base component (`xW`) and a lightweight residual component (`xA_i B_i`), allowing for effective disaggregation of the KV cache.
- Sharing the large `bCache` across all agents accessing the same context, while maintaining only small agent-specific `rCache`, drastically reduces redundant memory footprint without significant generation quality degradation.
- Operating system concepts like `fork` and `copy-on-write` can be adapted to efficiently manage the distinct lifecycles and sharing patterns of disaggregated KV cache components in multi-tenant LLM serving.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.