Reproduce “RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guannan Lai, Haoran Hu, Han-Jia Ye

RouteJudge evaluates routers through user preferences, while ORBIT provides offline development and submission workflow. Under a query, model pool, and budget, routers select models whose responses are compared anonymously, with votes attributed to participating routers. In 109 user-voted comparisons, RouterLLM-MF led Elo at 1278, while NIRT-Router led clear-winner win rate at 80.00%, illustrating metric-dependent results rather than a definitive ranking.

Key insights

  • User preference for LLM responses is pluralistic and context-dependent, often diverging from static offline metrics, thus necessitating online, preference-aware evaluation for true performance assessment.
  • A standardized, modular framework like ORBIT is essential for promoting reproducible research and streamlining the integration of diverse routing algorithms, fostering an open development ecosystem.
  • Effective LLM routing is a multi-objective decision problem, requiring trade-offs between quality, cost, latency, and nuanced user preferences that vary by task and budget, which fixed-target offline methods fail to capture.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.