Reproduce “Stabilizing Reinforcement Learning for Diffusion Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jianyuan Zhong, Kaibo Wang, Ding Ding, Zijin Feng, Haoli Bai, Yang Xiang, Jiacheng Sun, Qiang Xu

StableDRL, a reformulated Group Relative Policy Optimization, provides stable, full-parameter reinforcement learning for discrete diffusion language models by addressing training instabilities that lead to reward collapse. This methodology enables dLLMs to achieve state-of-the-art reasoning performance, for instance, securing 41.8% average accuracy on MATH500, which is a +5.2% improvement over existing methods.

Key insights

  • The instability in dLLM Reinforcement Learning stems from a self-reinforcing loop: noisy importance ratio estimates cause unbounded gradient spikes, leading to policy drift, which further amplifies ratio noise.
  • GRPO's conditional clipping and fixed group-size normalization, while effective for autoregressive models, become design flaws when faced with the high variance of estimated importance ratios in dLLMs.
  • Unconditional clipping prevents unbounded spikes but is insufficient alone; self-normalization is crucial to constrain update magnitudes within the convex hull of per-sample gradients, structurally breaking the instability loop.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.