Reproduce “Cartridges: Lightweight and general-purpose long context representations via self-study” on OpenResearch

About this paper

Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra and 3 more

The authors introduce Cartridges, compact document-specific key-value caches trained offline while keeping the language model frozen. Self-study generates synthetic conversations about document chunks and trains the cache to match the model’s next-token distributions when those chunks are in context. Across long-context tasks, self-study Cartridges use 38.6× less KV-cache memory and provide 26.4× higher peak serving throughput than ICL, while matching ICL quality.

Key insights

• Offloading context preparation to an offline training phase ("baking in" context) dramatically improves online serving efficiency by amortizing costs.
• A self-supervised training approach using synthetic data generation and context distillation effectively transfers long-context understanding to compact representations without sacrificing generality.
• Prefix-tuning is a robust parameterization for Cartridges, especially for out-of-domain generalization, and independently trained Cartridges can be composed at inference time.

Summary by alphaXiv.