Reproduce “sDPO: Don't Use Your Data All at Once” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, Chanjun Park
Stepwise DPO (sDPO) enhances large language model alignment by incrementally utilizing preference datasets, allowing previous alignment steps to establish a progressively stronger reference model. This method improves performance over standard DPO and enables smaller models to rival larger ones across key benchmarks.
Key insights
- The quality and alignment of the reference model are empirically crucial for effective Direct Preference Optimization.
- Internally generating progressively more aligned reference models via iterative DPO steps avoids the complexities and risks associated with external, pre-aligned references.
- Stepwise application of DPO with an easy-to-hard data partitioning strategy creates a curriculum learning effect, enhancing alignment stability and depth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.