Reproduce “CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control” on OpenResearch

About this paper

Qiaoling Chen, Zhisheng Ye, Tian Tang, Peng Sun, Boyu Tian, Guoteng Wang, Shenggui Li, Yonggang Wen and 2 more

CONCUR regulates which agents may generate from GPU cache feedback, limiting cache overcommitment during offline batches of tool-using agents. In one DeepSeek-V3 trace, middle-phase thrashing lasted over 90% of execution time, and recomputation accounted for 49.1% of latency. For Qwen3-32B with batch 256, it completed in 846 ms, versus 2,527 ms for uncontrolled SGLang in that configuration.

Key insights

• "Middle-phase thrashing" is a distinct and dominant performance bottleneck in agentic LLM batch inference, characterized by collapsing cache efficiency despite sustained high memory usage.
• Proactive, agent-level admission control is essential for agentic workloads because it addresses the unique challenge of semantically critical, temporarily inactive agent contexts that are mishandled by reactive, request-level eviction policies.
• Congestion control principles from distributed systems, specifically AIMD, can be effectively adapted to manage GPU KV cache resources for LLM serving, preventing performance collapse before it occurs.

Summary by alphaXiv.