Reproduce “Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning” on OpenResearch

About this paper

Zhiyin Yu, Bo Zhang, Qibin Hou, Zhonghai Wu, Xiao Luo, Lei Bai

Researchers from Peking University and collaborators developed EasyRL, a data-efficient reinforcement learning framework enabling Large Language Models (LLMs) to self-evolve their reasoning capabilities from limited easy labeled data and extensive difficult unlabeled data. The framework achieved up to a 12.1% performance improvement on mathematical reasoning tasks and demonstrated strong out-of-domain generalization to scientific reasoning, notably outperforming supervised methods trained on 100% labeled data while using only 10% easy labeled data.

Key insights

• LLMs can effectively self-evolve their reasoning capabilities from a small set of easy labeled data by progressively learning from increasingly difficult unlabeled examples, mirroring human cognitive development.
• A combined strategy of consistency-based selection and reflection-based resolution enables reliable pseudo-label generation for unlabeled data, effectively identifying and leveraging samples within the model's 'Zone of Proximal Development'.
• Iterative self-training that dynamically exposes the model to progressively more challenging pseudo-labeled samples is critical for sustained performance improvement and robust generalization.

Summary by alphaXiv.