Reproduce “Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minjae Lee, Kyunghyun Cho, Sangdon Park
MAESTRO trains a single preference-conditioned model to support user-selected trade-offs among objectives. It places preference weights in the prompt and adaptively shifts Dirichlet sampling toward preferences the policy serves poorly. On BeaverTails Harmless–Helpful, it achieved mean hypervolume 10.590 ± 0.221 over five seeds, versus 9.475 ± 0.107 for Bone Soups. Results use frozen reward models, not human judgments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.