Reproduce “Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhaochun Li, Chen Wang, Jionghao Bai, Shisheng Cui, Ge Lan, Zhou Zhao, Yue Wang
The paper introduces Distribution-Centric Policy Optimization (DCPO), an algorithm that redefines exploration in reinforcement learning for large language models from a sample-centric to a distribution-centric perspective. It consistently improves LLM performance on challenging reasoning benchmarks by managing the exploration-exploitation trade-off through a principled, on-policy entropy regulation mechanism.
Key insights
- The ability of a policy to resist entropy collapse and sustain exploration is primarily governed by the distribution under which the expected gradient is evaluated, rather than the diversity of individual sampled instances.
- By incorporating a REINFORCE policy gradient as a regularization term and using double importance sampling, an on-policy algorithm can effectively guide exploration towards a higher-entropy distribution.
- The work introduces a "Precision-Prediction (PP) Trade-off" to analyze the balance between precise sampling from an exploratory distribution and predicting an exploratory distribution via importance sampling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.