Reproduce “Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hao Xiang, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Ben He, Le Sun and 2 more
Self-Steering Optimization (SSO), developed by the Chinese Information Processing Laboratory and Alibaba Group, enables large language models to autonomously generate high-quality, on-policy preference data for alignment. This method consistently improves LLM performance on subjective and objective benchmarks, often outperforming or complementing human-annotated datasets.
Key insights
- Optimal preference data for LLM alignment must be both highly accurate (chosen response genuinely better than rejected) and on-policy (especially for rejected responses, ensuring they are probable outputs of the current model).
- A self-steering loop can be established by optimizing the policy model itself to act as a data generator, using specific loss functions to control the quality and distribution of generated preference pairs.
- Clever formulation of the 'bad' response loss (`L-`) by leveraging an 'original response' (`yo`) ensures that rejected responses remain on-policy, allowing for effective negative gradient application during optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.