Reproduce “Decoding-Time Language Model Alignment with Multiple Objectives” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, Simon S. Du
An algorithm named Multi-Objective Decoding (MOD) aligns language models to multiple objectives dynamically at inference time without retraining, by combining independently trained single-objective models' predictive distributions. It surpasses parameter-merging methods by up to 12.8% relative reward improvement on multi-objective tasks and can combine diverse models to improve metrics like coding ability and safety while reducing toxicity to nearly 0%.
Key insights
- A critical bijection exists between optimal policies and rewards under f-divergence regularization, enabling a principled approach to multi-objective alignment.
- Parameter-merging techniques are theoretically sub-optimal for f-divergence regularized policies, leading to "reward mis-specification."
- The framework provides precise control over objective trade-offs, including the ability to use negative weightings to steer away from specific traits, and unifies various decoding-time alignment methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.