Reproduce “WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse” on OpenResearch

About this paper

Aaryam Sharma

WavePP coordinates prefix-cache reuse, suffix-memory reservations, and chunk scheduling asynchronously so pipeline-parallel LLM prefill can keep GPUs supplied with work while requests are prepared. Across tested models and configurations, it often improves throughput over existing runtimes, with the largest gains under high prefix reuse and concurrency, though some low-concurrency and tail-latency cases regress.

Key insights

• Pipeline throughput can be constrained by distributed cache admission—not just GPU execution or chunk scheduling—when reusable state must be validated across stages.
• Protecting a common prefix and reserving suffix capacity together helps prevent cache eviction or memory pressure from invalidating admitted work.
• Smaller scheduling waves can improve pipeline utilization when little uncached work remains, but adaptive chunking is workload-sensitive and can reduce throughput in some settings.

Summary by alphaXiv.