Reproduce “Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinze Shi, Litian Zhang, Zhenjie Su, Zelong Yu, Yuting Zhang, Qihua Feng, Zifeng Kang, Chaozhuo Li and 1 more
The authors test whether comparative feedback teaches a language world model outcome frequencies in a simulated robot-push task. Event Bradley–Terry and judge-aware Full Channel training use identical comparisons and the same starting model. At a budget of 64 comparisons, Event Bradley–Terry assigns 39.0 percentage points to reference-zero outcomes, versus 0.5 for Full Channel, showing distortion in this tested setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.