Reproduce “Batch Query Processing and Optimization for Agentic Workflows” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junyi Shen, Noppanat Wadlom, Yao Lu
Halo makes batches of agent workflows schedulable as dependency graphs containing GPU LLM and CPU tool nodes. Its optimizer uses profiled costs and worker and cache state, while the runtime batches inference, coalesces identical tool calls, and overlaps eligible CPU and GPU work. On W1, 1,024 queries finished in 676 seconds versus 1,595 for OpWise, a 2.4× speedup on one multi-GPU server.
Key insights
- Treating agentic workflows as structured query-plan DAGs enables the system to identify and exploit opportunities for shared computation and dependencies across concurrent tasks.
- Global, dependency-aware placement and scheduling that explicitly consider heterogeneous CPU and GPU resources are crucial for maximizing hardware efficiency and minimizing pipeline stalls.
- Integrating LLM-specific optimizations, such as KV-cache reuse and adaptive batching, directly into a unified DAG scheduler significantly improves overall system throughput and reduces latency without compromising output quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.