Reproduce “Eliciting Human Preferences with Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Belinda Z. Li, Alex Tamkin, Noah Goodman, Jacob Andreas

GATE lets GPT-4 elicit preferences through generated examples or questions, then predict users’ judgments on held-out cases. In experiments with three bounded tasks, its interactive methods outperformed user-written prompts in 6 of 10 settings on the absolute p(correct) measure and 7 of 10 on time-based AUC. Statistical significance was established in content recommendation and email, not moral reasoning. This evidence does not establish broad alignment.

Key insights

  • Generative open-ended questions often perform strongly in eliciting preferences, outperforming other GATE methods in domains like moral reasoning and email validation.
  • Interactive elicitation methods, particularly those within GATE, generally reduce mental demand for users and help them discover unconsidered aspects of their preferences.
  • Model-model experiments can predict human results in certain domains (content recommendation, email validation), suggesting their utility for initial validation, though not universally.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.