Reproduce “Rethinking LLM Ensembling from the Perspective of Mixture Models” on OpenResearch

About this paper

Jiale Fu, Yuchu Jiang, Peijun Wu, Chonghan Liu, Joey Tianyi Zhou, Xu Yang

Researchers propose an efficient method for ensembling Large Language Models by reinterpreting the ensemble mechanism as a mixture model, named Mixture-model-like Ensemble (ME). This approach achieves generation quality comparable to traditional ensembling while requiring only a single model's forward pass per token, leading to substantial inference speedups of 1.78x to 2.68x over conventional methods across various datasets and hardware.

Key insights

• Sampling from a weighted ensemble distribution is statistically equivalent to a two-step stochastic process of randomly selecting a model and then sampling from that single model's distribution.
• This reinterpretation allows for maintaining ensemble performance while drastically reducing computational cost to that of a single model's forward pass per token.
• LLM ensembling, when viewed through the lens of mixture models and random selection, can be seen as the most basic form of token-level routing, conceptually linking it to Mixture-of-Experts strategies.

Summary by alphaXiv.