Reproduce “Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows” on OpenResearch

About this paper

Bochao Feng, Jianjiang Li, Haojie Wang, Lin Qiao, Yinghui Li, Yukun Yan, Jidong Zhai

The authors introduce a workflow-level scheduler that chooses which ready LLM turn to release and when, tracking committed work and prioritizing turns by tail risk. On SWE-bench with Llama-3.3-70B and four A100 GPUs at 0.9 workflows/s, it reduced P95 workflow flow time from 1,986.3 to 568.0 seconds, a 71.4% reduction; gains were negligible at light load.

Key insights

• Tail latency can be improved at the workflow-runtime boundary without requiring changes to the inference engine: controlling admission preserves the ability to react to changing workflow urgency.
• Tail-aware ordering is the main source of improvement, while adaptive budgeting adds a smaller but consistent benefit by preventing excessive queued work from accumulating inside the engine.
• The approach remains close to eager release under light load, suggesting that decoupled admission primarily helps when contention makes internal engine queues difficult to manage.

Summary by alphaXiv.