Reproduce “FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yitong Li, Junsong Chen, Shuchen Xue, Pengcuo Zeren, Siyuan Fu, Dinghao Yang, Yangyang Tang, Junjie Bai and 3 more

Sol-RL introduces an FP4-empowered, two-stage framework for Diffusion Reinforcement Learning, efficiently scaling rollouts by decoupling low-precision exploration from high-fidelity optimization. This approach achieved up to 4.64x faster training convergence and superior alignment performance across various foundation models and reward metrics, while preserving training fidelity.

Key insights

  • Decoupling the high-throughput exploration phase from the high-fidelity policy optimization phase effectively resolves the trade-off between efficiency and training integrity in diffusion RL.
  • NVFP4 quantization reliably preserves the relative intra-group ranking of samples, especially for extreme top/bottom candidates, validating its use as a proxy for efficient contrastive sample selection.
  • Scaling the FP4 exploration pool significantly broadens the search space for high-contrastive samples, leading to improved final alignment even when only a subset is regenerated for BF16 training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.