Reproduce “Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints” on OpenResearch

About this paper

Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu, Zhengze Zhou, Zhipeng Wang, Rahul Mazumder

Researchers from MIT and others propose a batch-level, resource-aware routing framework for Large Language Models (LLMs) that explicitly manages monetary cost, GPU capacity, and model concurrency limits. It leverages integer linear programming with robust optimization to handle performance prediction uncertainty and includes an offline procedure for optimal model instance allocation, achieving up to 24% performance improvement under adversarial batching compared to per-query methods.

Key insights

• Batch-level optimization consistently outperforms per-query routing, particularly under adversarial conditions, by effectively managing cost variance and ensuring resource adherence.
• Incorporating robust optimization leads to more reliable routing decisions by accounting for predictive uncertainty, often favoring LLMs with lower estimated performance variance.
• Strategic offline allocation of model instances significantly improves performance and resource utilization compared to uniform allocation, especially when cost budgets are tight.

Summary by alphaXiv.