Reproduce “A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancement” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shengji Tang, Jianjian Cao, Weihao Lin, Jiale Hong, Bo Zhang, Shuyue Hu, Lei Bai, Tao Chen and 2 more

A scalable multi-LLM collaboration system (SMCS) integrates retrieval-based prior LLM selection with exploration-exploitation-driven posterior response enhancement. This system achieved an average accuracy of 76.78% across eight benchmarks, outperforming leading closed-source models such as GPT-4.1 by 5.36% and demonstrating significant scalability and efficiency with open-source LLMs.

Key insights

  • Dynamically selecting LLMs based on their fine-grained, instance-level performance on similar past questions proves more scalable and effective than static category-based or router-trained selection methods.
  • Combining exploration through diverse response aggregation with exploitation via hybrid scoring significantly refines outputs and enhances overall quality beyond individual LLM capabilities.
  • Orchestrating a diverse set of mid-sized open-source LLMs through integrated prior selection and posterior enhancement allows their collective intelligence to surpass individual state-of-the-art closed-source models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.