Reproduce “Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries” on OpenResearch

About this paper

Junhyuck Kim, Jongho Park, Jaewoong Cho, Dimitris Papailiopoulos

Researchers from KRAFTON and academic institutions developed LEXICO, a method for extreme compression of the KV cache in Large Language Models using sparse coding with universal dictionaries. This approach reduces the KV cache size to 15-20% of the original, maintaining performance on tasks like LongBench and GSM8K, enabling more memory-efficient LLM inference.

Key insights

• Key and value vectors in transformer models exhibit a low-dimensional structure, suggesting they can be efficiently represented as sparse combinations of a few basis vectors.
• Universal dictionaries, trained once on a general dataset, can effectively represent KV cache states across diverse inputs and tasks.
• Maintaining a small uncompressed buffer of recent tokens significantly enhances performance by preserving critical immediate context, while still enabling high overall compression.

Summary by alphaXiv.