Reproduce “QuRL: Efficient Reinforcement Learning with Quantized Rollout” on OpenResearch

About this paper

Yuhang Li, Reena Elangovan, Xin Dong, Priyadarshini Panda, Brucek Khailany

Researchers from Yale University and NVIDIA Research present QURL, a method for efficient reinforcement learning (RL) with Large Language Models (LLMs) that accelerates the rollout phase through quantization. It introduces Adaptive Clipping Range (ACR) and Update-Aware Quantization (UAQ) to enable stable training, achieving 20% to 90% faster rollout throughput while matching or improving accuracy on reasoning tasks.

Key insights

• The instability caused by divergent behavior (quantized) and target (full-precision) policies in RL can be effectively managed by dynamically adjusting the clipping range in the PPO objective.
• Small weight changes characteristic of RL updates are often masked by quantization granularity, requiring a specialized technique like UAQ to ensure these updates are captured by the quantized model.
• By strategically scaling weights before quantization, it is possible to simultaneously reduce quantization error and amplify gradients, leading to an improved ratio between weight updates and quantization noise.

Summary by alphaXiv.