Reproduce “Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chitral Patil

A new methodology for LLM infrastructure cost estimation accounts for variable GPU utilization, demonstrating that effective cost per million tokens can vary by up to 36.3 times based on the offered request rate. This work includes an open-source tool for real-time cost measurement and provides corrected break-even analyses for self-hosting versus commercial APIs.

Key insights

  • GPU utilization is an output of the system's interaction with offered load, not a fixed input, fundamentally altering LLM inference cost economics and making underutilization the dominant cost factor at low loads.
  • Achieving production-grade latency Service Level Objectives (SLOs) necessitates operating below maximum theoretical throughput, incurring an "SLA premium" that can increase effective costs by 1.13x to 1.91x over unconstrained saturation costs.
  • Corrected crossover analyses, incorporating utilization dynamics, demonstrate that self-hosting can be more expensive than commercial APIs at low traffic volumes, challenging naive cost assumptions and requiring higher traffic thresholds for cost-effectiveness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.