Reproduce “SemPIC: Learning Semantic Position-Independent KV Caches” on OpenResearch

About this paper

Hui Xie, Peng Xiao, Yutong Deng, Shuoran Dou, Jian Yang, Jinyang Guo

SemPIC introduces a method for learning semantic position-independent KV caches for large language models, enabling efficient reuse of contextual information in long-context workloads. It achieves this by offline compilation of entire document KV states, adapted via LoRA and behavioral distillation, which improves task-level micro-F1 performance from 0.53 to 0.60 compared to prior methods while maintaining an unchanged online inference interface.

Key insights

• Full quality recovery in position-independent caching requires document-wide adaptation of KV representations, not just boundary-specific adjustments, to resolve contextual incompleteness.
• A knowledge distillation approach can effectively train an offline 'Writer' to generate context-aware KV caches that an *unchanged* pretrained LLM 'Reader' can seamlessly integrate during inference.
• Efficient training for such systems on long documents necessitates specialized memory management techniques, like KV Gradient Checkpointing, to handle the dual memory demands of both full-context KVs and Writer intermediates.

Summary by alphaXiv.