Reproduce “Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jingjie Ning, Xiaochuan Li, Ji Zeng, Hao Kang, Chenyan Xiong

Researchers at Carnegie Mellon University introduce an "auto research" framework where specialist language agents autonomously modify, run, and refine machine learning training recipes based on externally measured feedback. This system achieved quantifiable improvements across diverse tasks, including a 0.81% reduction in validation bpb, a 38.7% gain in CORE score, and a 4.59% speedup in training wallclock time, all without human intervention during the search loop.

Key insights

  • Lineage feedback is critical for agents to convert diverse evaluator outcomes, including various types of failures (e.g., size blocks, budget overruns, accuracy-gate misses), into concrete, program-level recipe modifications.
  • Role partitioning among specialist agents, combined with lineage, significantly enhances proposal diversity and boundary adherence, outperforming generic multi-agent or single-agent approaches.
  • The autonomous loop demonstrates capabilities in compositional improvements, such as combining, transferring, and repairing known techniques under external feedback and respecting constraints.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.