Reproduce “Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiyan Fu, Wei Liu
Researchers Xiyan Fu and Wei Liu introduced an outcome-level reinforcement learning paradigm for compositional generalization, which consistently improved performance over token-level supervised fine-tuning. Their method leverages direct feedback on complete output sequences, leading to better generalization across diverse benchmarks and mitigating overfitting to training patterns.
Key insights
- Outcome-level optimization is more effective for learning compositional behaviors than token-level imitation, as it directly optimizes for the global correctness of generated sequences.
- Reinforcement learning significantly reshapes the model's output distribution, mitigating the tendency of supervised fine-tuning to overfit and reproduce frequent training patterns.
- Even a simple binary exact match reward can be sufficient to guide effective policy optimization for compositional generalization, suggesting that highly complex reward functions are not always necessary for substantial gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.