Reproduce “Training Language Models to Win Debates with Self-Play Improves Judge Accuracy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samuel Arnesen, David Rein, Julian Michael
Research from New York University demonstrated that training language models to win debates through a self-play DPO approach improved AI judge accuracy by 4 percentage points, a gain not observed with non-adversarial "consultancy" training. This highlights the benefits of adversarial interaction for truth-seeking AI.
Key insights
- Training models to win debates fosters behaviors useful for truth-seeking, leading to more accurate judgments by an AI judge.
- Debate training incentivizes models to provide more evidence-based arguments and develop argumentation strategies that are transferable to different judges.
- The judge's access to arguments from both sides and their comparative presentation contributes more to judge accuracy in this setting than explicit refutation by debaters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.