Reproduce “Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Saicharan Kolluru

An empirical comparison of vLLM and HuggingFace TGI evaluates their performance for Large Language Model inference serving across LLaMA-2 models. The analysis reveals vLLM offers higher throughput and GPU utilization with less memory via PagedAttention, while TGI provides lower Time-to-First-Token and more consistent latency for individual requests.

Key insights

  • vLLM's PagedAttention memory management and continuous batching enable superior GPU utilization, higher throughput, and a reduced memory footprint by eliminating KV cache fragmentation and sharing memory.
  • TGI prioritizes lower Time-to-First-Token (TTFT) and more predictable individual request latency due to its batch-level dynamic batching strategy, making it better suited for interactive applications.
  • The optimal choice of framework depends on specific application requirements, with vLLM excelling in throughput-intensive scenarios and TGI being more suitable for latency-sensitive, interactive use cases.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.