Reproduce “PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving” on OpenResearch

About this paper

Omkar Shewale, Deepak Kumar, Divakar Kumar Yadav

PrefixBench-H100 measures prefix-cache reuse and time-to-first-token (TTFT) for vLLM 0.21.0 and TensorRT-LLM 1.2.1 on an NVIDIA H100 NVL using synthetic and RAG-style workloads. Long shared prefixes reduced median TTFT by up to 6.5×, while concurrency and arrival patterns produced larger runtime differences than cache hit rates: vLLM favored steady traffic and throughput, whereas TensorRT-LLM achieved lower median TTFT in some highly bursty workloads.

Key insights

• Prefix caching is highly effective when the working set fits in memory: an 8,192-token shared prefix reduced vLLM p50 TTFT from 345.1 ms to 53.0 ms and TensorRT-LLM p50 TTFT from 306.1 ms to 58.2 ms.
• The two runtimes achieved similar cache hit rates, so their latency differences were mainly associated with scheduling and execution behavior rather than prefix-cache effectiveness.
• Arrival pattern and concurrency materially change runtime rankings: vLLM performed better for steady RAG-style traffic and aggregate throughput, while TensorRT-LLM had lower p50 TTFT at the highest burst concurrency but worse p95 latency in fixed-rate tests.

Summary by alphaXiv.