Reproduce “To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Taehyung Kim, Jongeun Choi

CurriPO (Probe-Driven Reward Traversal) introduces a method that utilizes the diversity in human preferences to create an adaptive curriculum for policy optimization in personalized AI alignment. This approach substantially improves overall population satisfaction, achieving 1.2–2.1 times the satisfaction of baseline methods, and significantly reduces training time while effectively serving users with conventionally difficult-to-satisfy preferences.

Key insights

  • The diversity of user preferences in a multi-user setting can be strategically leveraged to form an adaptive curriculum for policy optimization, improving alignment efficiency and coverage.
  • Optimization difficulty is a distinct challenge from reward model accuracy, and addressing it through curriculum learning is crucial for comprehensive and fair AI alignment across heterogeneous populations.
  • Mechanisms like branching and reusing previously optimized policy checkpoints are vital for efficiently exploring the policy space and effectively serving users with a wide range of heterogeneous, and potentially difficult, preferences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.