Reproduce “Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning” on OpenResearch

About this paper

Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, Min Chen

CORY introduces a method for fine-tuning large language models using a sequential cooperative multi-agent reinforcement learning framework, which enhances training stability and resistance to distribution collapse compared to conventional single-agent reinforcement learning approaches. It demonstrates improved policy optimality by balancing task reward maximization with maintaining the pre-trained model's distribution.

Key insights

• Viewing LLM fine-tuning with a KL penalty as a multi-objective optimization problem, CORY achieves a superior trade-off between maximizing task reward and minimizing KL divergence, approaching the Pareto frontier more closely.
• The sequential cooperative interaction and collective reward mechanism facilitate a form of bootstrapping and coevolution, allowing agents to mutually improve their policies and inherent resistance to distribution collapse.
• The framework's design, including knowledge transfer and role exchange, effectively addresses the instability and over-optimization issues commonly found in PPO-based LLM fine-tuning.

Summary by alphaXiv.