Reproduce “Weighted-Reward Preference Optimization for Implicit Model Fusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, Xiaojun Quan
Researchers from Sun Yat-sen University developed Weighted-Reward Preference Optimization (WRPO), a method for implicitly fusing heterogeneous large language models into a single target model. This approach effectively addresses distributional shifts during knowledge transfer, achieving superior performance against explicit fusion methods and even larger source models, while introducing only a 16% training overhead compared to DPO.
Key insights
- The dynamic weighting of rewards, controlled by the fusion coefficient α, effectively mitigates distributional deviations between heterogeneous source and target LLMs, enabling stable knowledge transfer.
- Both source-model-preferred responses (`y_w_s`) and target-model-preferred responses (`y_w_t`) are crucial for bridging the distribution gap and providing a robust preference signal.
- Implicit model fusion, when properly calibrated to handle heterogeneous data, can outperform explicit fusion methods and achieve strong performance without increasing inference-time computational costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.