Reproduce “Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benjamin Novotny, Doug Szeto, Chang Wang, Shicheng Chen, George Zhihua Yuan, Bob Kamma, Michael Ge, John Xin Zhou and 1 more
Cap-and-Track dispatches asynchronous LLM work by rejecting excess requests at capacity-constrained engines, routing them to a durable priority queue, and adjusting dispatch from capacity refusals. In replay, it delivered 108.5 requests per second per host at full capacity, close to 109.5 for a fixed rate, while refusal share was 3.8% versus 15.5%. It does not establish universal superiority.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.