Reproduce “Pretraining Language Models with Human Preferences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, Ethan Perez
Integrating human preference feedback directly into language model pretraining, especially through conditional training, demonstrably yields models better aligned with human values. This approach reduces undesirable outputs like toxicity by an order of magnitude while preserving general linguistic capabilities, surpassing the effectiveness of traditional post-hoc alignment methods.
Key insights
- Integrating human preferences directly into the pretraining phase is substantially more effective at achieving alignment than the conventional approach of MLE pretraining followed by post-hoc finetuning.
- Conditional training emerges as a Pareto-optimal PHF objective, demonstrating significant improvements in alignment while largely maintaining general language model capabilities across various benchmarks.
- PHF allows models to learn from a diverse dataset, including potentially imperfect content, by explicitly guiding the model not to imitate undesirable behaviors rather than simply removing problematic data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.