Reproduce “Distributed Mixture-of-Agents for Edge Inference with Large Language Models” on OpenResearch

About this paper

Purbesh Mitra, Priyanka Kaswan, Sennur Ulukus

A distributed Mixture-of-Agents (MoA) framework for Large Language Model (LLM) inference on edge devices was developed by researchers at the University of Maryland and Princeton University, establishing queuing stability conditions and improving response accuracy by up to 55.55% on an AlpacaEval 2.0 subset when using diverse LLMs, though at the cost of increased latency and queue sizes.

Key insights

• The queuing stability of distributed MoA systems on edge devices is directly quantifiable and constrained by the product of the average LLM inference time, the effective prompt arrival rate, and the MoA configuration parameters, summarized by the condition ((k + 1)M + 1) < 1.
• A fundamental trade-off exists: achieving higher LLM response accuracy through more extensive collaboration (increased layers 'M' and proposers 'k') consistently results in greater system latency and larger average queue sizes.
• Integrating a diverse set of Large Language Models within the distributed MoA framework can further enhance overall response accuracy compared to using homogeneous LLM configurations.

Summary by alphaXiv.