Reproduce “Strategyproof Reinforcement Learning from Human Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomas Kleine Buening, Jiarui Gan, Debmalya Mandal, Marta Kwiatkowska
Researchers from ETH Zurich and the University of Oxford introduce a formal framework for Reinforcement Learning from Human Feedback (RLHF) that accounts for strategic human labelers who might misreport preferences. They propose the Pessimistic Median of MLEs (Pessimistic MoMLEs) algorithm, which achieves approximate strategyproofness and converges to optimal social welfare, while also identifying a fundamental trade-off between incentive alignment and policy alignment in multi-labeler settings.
Key insights
- Existing pluralistic RLHF methods (e.g., Pessimistic Social Welfare) are not strategyproof, allowing a single strategic labeler to cause arbitrary degradation in social welfare.
- There is an inherent trade-off in strategyproof RLHF: any such algorithm incurs a worst-case constant suboptimality of at least (k-1)/k and an approximation ratio no better than 1/k, where k is the number of labelers.
- Approximate strategyproofness and convergence to optimal social welfare can be achieved by optimizing a pessimistic median return, balancing robustness against strategic behavior with policy alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.