Reproduce “Near-Optimal Online Deployment and Routing for Streaming LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaoang Li, Jian Li
Researchers from Stony Brook University developed StageRoute, a hierarchical algorithm for near-optimal online deployment and routing of streaming large language models, addressing dynamic model availability, concurrency caps, and cost/throughput limits. The algorithm achieves an expected regret bounded by O(T^(2/3)) and empirically outperforms baseline strategies while adhering to operational constraints.
Key insights
- The problem is formalized as a unique online deployment and routing challenge, featuring dynamic model arrivals, a hard concurrency cap, one-time deployment costs, and a long-term budget, with decisions made at two distinct timescales.
- The regret analysis distinguishes between a statistical learning cost for routing within the deployed set and a structural model-discovery bottleneck, quantifying the challenge of exploring new models under limited deployment slots.
- A hierarchical optimization approach using UCBs for rewards and LCBs for costs in both deployment (MIP) and routing (LP) phases enables effective balancing of exploration and exploitation under strict operational constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.