Reproduce “Style Transfer with Multi-iteration Preference Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuai Liu, Jonathan May
The STAMP framework integrates a multi-iteration preference optimization process with dynamic weighted reward aggregation and hope-and-fear sampling to enhance text style transfer. It demonstrated improved aggregated performance, outperforming previous state-of-the-art methods on both automatic and human evaluations across CDS and GYAFC datasets.
Key insights
- Iterative optimization, adapted from statistical machine translation (SMT) tuning, effectively refines generative language models for text style transfer, enhancing performance over single-pass approaches.
- Dynamically adjusting reward weights during multi-objective optimization is crucial for maintaining balance across fluency, meaning similarity, and target style strength, preventing individual objective degradation.
- Hope-and-fear sampling, which selects 'winning' and 'losing' examples based on combined reward and model scores, provides more informative contrastive pairs for preference optimization than simpler sampling methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.