Reproduce “DiRL: An Efficient Post-Training Framework for Diffusion Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ying Zhu, Jiaxin Wan, Xiaoran Liu, Siyang He, Qiqi Wang, Xu Guo, Tianyi Liang, Zengfeng Huang and 2 more

DiRL introduces an efficient post-training framework for Diffusion Language Models (dLLMs), leveraging engineering optimizations and an unbiased Group Relative Policy Optimization (DiPO) algorithm. This framework enables dLLMs to achieve state-of-the-art performance in mathematical reasoning, often outperforming larger auto-regressive models.

Key insights

  • Engineering optimizations, particularly FlexAttention and LMDeploy's online updates, are critical for scalable dLLM post-training, achieving a 2.55x throughput improvement.
  • DiRL demonstrates that dLLMs, with proper and efficient post-training, can achieve and surpass the reasoning capabilities of larger Auto-Regressive models on complex tasks.
  • DiPO successfully adapts Group Relative Policy Optimization for dLLMs, providing the first unbiased implementation that resolves logit computation challenges and enables effective policy alignment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.