Reproduce “DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liang Zhu, Feiteng Fang, Yuelin Bai, Longze Chen, Zhexiang Zhang, Minghuan Tan, Min Yang
DEFT (Distribution-guided Efficient Fine-Tuning) proposes a framework to enhance large language model alignment by deriving a discrepancy distribution from human preference data and applying a novel distribution reward. This approach efficiently filters high-quality training data and guides fine-tuning, reducing training time from approximately 48 hours to 3 hours while improving alignment performance and, for some methods, boosting generalization capabilities on benchmarks like MT-Bench.
Key insights
- Aligning the overall output distribution of a model (dubbed the 'superficial alignment hypothesis') can lead to more efficient and effective preference learning than token-level reinforcement.
- Responses that are most challenging for an unaligned model to generate correctly (those yielding the lowest distribution rewards) contain the most informative preference content for data filtering.
- The distributional guidance mechanism, using Q_diff and R_Q, provides a robust signal to direct the model towards desired token distributions, allowing for monotonic increases in reward during training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.