Reproduce “Post-Training with Policy Gradients: Optimality and the Base Model Barrier” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alireza Mousavi-Hosseini, Murat A. Erdogdu
Researchers from the University of Toronto and the Vector Institute theoretically analyzed policy gradient (PG) post-training for linear autoregressive models, revealing that outcome reward models (ORM) face a "base model barrier" requiring exponentially more reward queries with sequence length, while process reward models (PRM) overcome this by enabling linear scaling. This work establishes minimax optimality for PG variants and provides theoretical justification for PRM's efficiency.
Key insights
- Outcome reward models (ORM) lead to a fundamental "base model barrier" for PG post-training, where achieving significant improvements beyond the base model's initial support requires a number of reward queries that scales exponentially with the sequence length $N$.
- Process reward models (PRM), by providing feedback at each token generation step, can effectively overcome the base model barrier, reducing the reward query complexity to scale linearly with the sequence length $N$.
- Adaptive learning rate variants of policy gradient (PG-OR and PG-PR), when combined with suitable exploration strategies, are shown to be minimax optimal (up to logarithmic factors) in both statistical and online learning settings, relative to the base model's Likelihood Quantile.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.