Reproduce “Orchestrating Intelligence: Confidence-Aware Routing for Efficient Multi-Agent Collaboration across Multi-Scale Models” on OpenResearch

About this paper

Jingbo Wang, Sendong Zhao, Jiatong Liu, Haochun Wang, Wanting Li, Bing Qin, Ting Liu

OI-MAS selects agent roles and model scales per reasoning turn, adapting assignments as context changes. Its confidence signal adjusts the cost penalty, allowing model capacity or richer role combinations when warranted, but does not establish correctness. Across five tasks, it averages 78.23 versus MasRouter’s 70.55, beating it on four and tying on GSM8K; results are limited to benchmarks and setup.

Key insights

• Jointly selecting agent roles and dynamically assigning multi-scale LLM backbones at each step of the reasoning process, rather than static assignment, yields superior performance and efficiency.
• Model confidence, measured by average token log-probability, effectively indicates the complexity of a reasoning state and can guide the adaptive allocation of computational resources.
• A reinforcement learning framework that penalizes cost based on confidence allows the system to balance performance and efficiency by escalating to larger models only when task complexity warrants it.

Summary by alphaXiv.