Reproduce “TokenRouter: Efficient Serving System for Token-Level LLM Routing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang
TokenRouter serves inference with per-token routing between language models, using asynchronous subservers and delayed batch admission. Developers define routing and state handoffs; each model server buffers arrivals and uses a throughput model to select a threshold. On R2R’s AIME setup at concurrency 4, it reports 244.56 generated tokens/s versus 89.62 for official R2R, a serving-efficiency result under routing assumptions.
Key insights
- Token-level routing efficiency depends not only on the routing policy but also on how serving handles model switches, batching, and cache state.
- Asynchronous execution removes cross-model waiting, while delayed batching can rebuild larger model-local batches from irregular arrivals.
- Scheduling gains are most pronounced under concurrency; the reported scheduling components did not improve throughput in the concurrency-one ablation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.