Reproduce “Co-Evolving LLM Evaluators and Policies via DynamicRubric” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Beining Wang, Weihang Su, Hongtao Tian, Hao Kong, Tao Yang, Ting Yao, Qingyi Pan, Yueyue Wu and 3 more

A framework named DynamicRubric enables large language models and their evaluators to co-evolve, generating adaptive, fine-grained rubrics to provide effective supervision signals during post-training, leading to improved open-ended generation and successful deployment in a large-scale production environment. This method addresses the challenge of collapsed score gaps in advanced LLM feedback loops.

Key insights

  • The directional gain from policy updates is directly proportional to the relative evaluator score gap between responses, making the preservation of these gaps fundamental for effective optimization.
  • Response-set-conditioned evaluators possess a greater theoretical capacity to model and preserve fine-grained score gaps within candidate responses compared to prompt-only evaluators.
  • Continuous co-evolution between the evaluator and the policy, where each adapts to the other's current state, is essential to maintain effective supervision signals as LLM quality improves.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.