Reproduce “Binary Classifier Optimization for Large Language Model Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, Kyoung-Woon On
This paper from Kakao Corp. and LBOX introduces Binary Classifier Optimization (BCO), a method for aligning Large Language Models using only binary user feedback. BCO connects binary classification to Direct Preference Optimization (DPO) through a theoretically grounded reward shift technique, showing it achieves competitive alignment quality with DPO on paired preference data and notably outperforms DPO on datasets resembling real-world user feedback.
Key insights
- Minimizing the Binary Cross-Entropy (BCE) loss on desired/undesired completions acts as an upper bound for the Direct Preference Optimization (DPO) loss, providing a theoretical link between binary classification and preference-based alignment.
- A novel 'reward shift' technique, calculated as the average of implicit rewards from both positive and negative examples, significantly reduces the gap between BCE and DPO losses, enhancing alignment effectiveness and allowing for greater model divergence from the reference policy.
- BCO's objective function maintains more uniform gradients across different reward ranges compared to prior methods like KTO, leading to more stable and comprehensive learning from all data samples.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.