Reproduce “HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing” on OpenResearch

About this paper

Mao Lin, Xi Wang, Guilherme Cox, Dong Li, Hyeran Jeon

HybridGen introduces an efficient CPU-GPU hybrid attention framework for long-context LLM generative inference, designed to overcome memory and compute bottlenecks in heterogeneous systems. It achieved average speedups of 1.41x to 3.2x over state-of-the-art methods while maintaining output accuracy comparable to full attention.

Key insights

• Attention logit computation ($Q imes K^T$) is inherently parallel and can be decoupled, allowing concurrent processing of KV tokens residing in both CPU and GPU local memories.
• Inputs to consecutive transformer layers exhibit high similarity, enabling proactive CPU computation for subsequent layers to effectively overlap with the GPU's current layer processing.
• K and V vectors have distinct access patterns: K vectors are critical for CPU-side logit computation, while V vectors are primarily needed by the GPU for aggregation, allowing for optimized, semantic-aware placement in tiered memory.

Summary by alphaXiv.