Reproduce “Discovering Novel LLM Experts via Task-Capability Coevolution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Dai, Boris Meinardus, Ciaran Regan, Yingtao Tian, Yujin Tang
AC/DC coevolves models and synthetic tasks, merging existing models while a scientist LLM generates and validates challenges. It selects teams for Coverage, where a question counts if any member answers correctly. On Qwen2.5 7B, a three-model team gains 3.85 percentage points in average benchmark Coverage over a 72B baseline, using 21B parameters; results vary by family and team size.
Key insights
- Coevolution of LLMs and synthetic tasks autonomously discovers diverse, specialized experts without direct optimization on external benchmarks.
- Collectives of smaller, specialized LLMs can collectively achieve broader problem-solving coverage and parameter efficiency, surpassing single, larger models.
- Quality-Diversity (QD) selection and robust filtering mechanisms are crucial for guiding effective open-ended evolution and maintaining model coherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.