Reproduce “Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongqiu Ni, Jiabao Zhang, Guopeng Li, Zilong Wang, Ruiqi Wu, Chi Zhang, Haisheng Tan
Astraea is an LLM serving engine designed to efficiently manage multi-stage agentic workflows by employing a state-aware, hierarchical scheduling algorithm and adaptive Key-Value (KV) cache management. The system significantly reduces end-to-end Job Completion Time, outperforming existing baselines in various load and memory conditions.
Key insights
- Optimizing for the global end-to-end Job Completion Time (JCT) for multi-stage agentic workflows is critical, rather than just segment-level metrics.
- Proactive, state-aware scheduling that incorporates predictions of both computational and external API durations effectively mitigates Head-of-Line blocking and improves overall efficiency.
- Adaptive management of the Key-Value (KV) cache during I/O waits can dynamically balance the trade-off between minimizing recomputation latency and maximizing GPU memory utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.