Reproduce “Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection” on OpenResearch

About this paper

Guillem Ramírez, Alexandra Birch, Ivan Titov

Researchers at the University of Edinburgh developed Margin Sampling, a method that dynamically routes queries to either a smaller, cost-effective language model or a larger, more powerful one based on the smaller model's prediction uncertainty. This approach consistently outperformed other strategies in 25 out of 27 experimental setups across diverse tasks and model pairs, achieving better accuracy for the same computational cost.

Key insights

• The uncertainty in a smaller LLM's prediction, specifically the margin between its top two token probabilities, serves as an effective signal for dynamic model selection.
• Simple, non-parametric approaches based on inherent model properties can outperform more complex methods requiring additional training.
• Utilizing readily available output probabilities from existing LLMs allows for cost-efficient routing without introducing new architectural complexity or training overhead.

Summary by alphaXiv.