Reproduce “BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Carmen Hipolito Garcia, Menglin Xia, Laks V.S. Lakshmanan and 2 more
BEST-Route proposes an adaptive LLM routing framework that intelligently combines model selection with test-time optimal compute using best-of-n sampling. This approach enables substantial cost reductions in LLM inference, achieving up to 60% savings with less than a 1% quality degradation compared to using large, expensive models alone.
Key insights
- Combining dynamic LLM routing with adaptive test-time compute, specifically best-of-n sampling, allows smaller, cheaper models to achieve quality competitive with larger models for a broader range of queries.
- A compact, fine-tuned proxy reward model can effectively and efficiently rank and select the best response from multiple samples, significantly reducing the cost of best-of-n sampling.
- A multi-head router with a shared backbone efficiently predicts the optimal model and sampling rate for each query, ensuring minimal overhead and practical scalability across various model and compute configurations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.