Reproduce “CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Teng Pan, Yuchen Yan, Zixuan Wang, Ruiqing Zhang, Guiyang Hou, Wenqi Zhang, Weiming Lu, Jun Xiao and 1 more
CoVerRL introduces a co-evolutionary reinforcement learning framework that enables large language models to self-improve their reasoning without human labels, breaking the "consensus trap" prevalent in majority-voting approaches. This system achieves an average of 5.7% Acc.@final improvement across various models and benchmarks compared to a TTRL baseline, by mutually enhancing generator and verifier capabilities.
Key insights
- The "consensus trap" arises from relying solely on majority voting, as it reinforces self-consistency which can inadvertently lead to a collapse in output diversity and confident reinforcement of systematic errors.
- Generator and verifier capabilities can mutually bootstrap each other's improvement in a label-free setting, where noisy pseudo-labels train the verifier, and an improving verifier provides cleaner signals for the generator.
- Balanced training with contrastive examples and Answer-Anchored GRPO is crucial for preventing verifier collapse and ensuring stable co-evolution, enabling the verifier to learn reliable error detection without ground truth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.