Reproduce “LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yang Li

LLM Bandit introduces a cost-efficient framework for dynamic routing of queries to large language models based on user-defined performance-cost preferences. It significantly reduces operational costs, demonstrating up to a 27% cost reduction on the HELM-MMLU benchmark, and can incorporate new LLMs using only 20-50 evaluation prompts.

Key insights

  • A single preference-conditioned routing policy, trained with multi-objective reinforcement learning, can generalize across arbitrary sets of LLMs and diverse performance-cost preferences without specific retraining.
  • Model identity vectors, derived from Item Response Theory and pairwise comparisons, provide an efficient and generalizable representation of LLM capabilities.
  • New LLMs can be efficiently integrated ("cold start") into the routing system by deriving their identity vectors from as few as 20-50 targeted evaluation prompts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.