Reproduce “CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models” on OpenResearch

About this paper

Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan

CanvasAnneal introduces a training-time curriculum for reinforcement learning in discrete diffusion language models: partial teacher-generated reasoning traces initialize the diffusion canvas early in training, while the amount of visible guidance is progressively reduced and fully masked rollouts remain in the mix. On LLaDA-7B-A1B-Instruct, the method improves MATH500, Countdown, and Tau2 results and accelerates reward acquisition on xLAM and Countdown, though gains are task-dependent and do not consistently extend to GSM8K or all function-calling settings.

Key insights

• The diffusion canvas can serve as a curriculum variable: structured initial states make early exploration easier, while progressively removing the scaffold encourages the policy to recover native diffusion behavior.
• Contiguous suffix masking is more effective than the reported random-masking alternative, likely because it turns teacher guidance into a coherent continuation problem.
• Teacher-assisted exploration improves optimization speed and some downstream tasks, but its value depends strongly on the benchmark, reward structure, masking schedule, and balance between guided and fully masked rollouts.

Summary by alphaXiv.