Reproduce “Entropy-Preserving Reinforcement Learning” on OpenResearch

About this paper

Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, Philipp Krähenbühl

This research provides a formal analysis of policy entropy dynamics in reinforcement learning for large language models, revealing how numerical precision settings like bf16 quantization can distort entropy and lead to training instabilities. It introduces novel entropy-preserving algorithms, REPO and ADAPO, which improve performance on reasoning tasks and enhance models' trainability for sequential learning.

Key insights

• The trajectory of policy entropy during training is a critical indicator of learning effectiveness, with consistently low entropy correlating with lower final test accuracy.
• Subtle numerical precision issues, specifically bf16 quantization, can significantly distort entropy dynamics by introducing biases in probability ratio calculations, leading to inadvertent entropy collapse.
• Actively controlling policy entropy through methods like REPO and ADAPO not only improves performance on complex reasoning tasks but also preserves the model's trainability for sequential learning.

Summary by alphaXiv.