Reproduce “Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hsiang-Wei Huang, Junbin Lu, Kuang-Ming Chen, Jenq-Neng Hwang
Researchers from the University of Washington modeled multi-round peer review dynamics using LLM agents within an Elo-ranked system, demonstrating that this approach significantly improves Area Chair decision accuracy while revealing how reviewer agents strategically adapt their behaviors based on continuous feedback. The simulation highlights how an Elo system can stratify reviewer performance, rewarding quality and penalizing low effort, but also how visible scores can lead to strategic optimization of review style over substantive quality.
Key insights
- Elo-based feedback rapidly stratifies reviewer performance over time, clearly differentiating high- and low-performing agents, systematically rewarding 'Expert' personas and consistently penalizing 'Skimmer' (low-effort) personas.
- Making Elo ratings visible to reviewers (Full Access) incentivizes strategic adaptation, leading 'Critic' and 'Bluffer' personas to adjust their review styles (e.g., tone, selectivity) to improve their rank, rather than fundamentally enhancing substantive quality.
- While an Elo-ranked system significantly improves Area Chair decision accuracy, full transparency introduces a trade-off where strategic reviewer adaptation can lead to higher precision but lower recall, possibly at the cost of diverse review perspectives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.