Reproduce “Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Julianna Piskorz, Cristina Pinneri, Alvaro Correia, Motasem Alfarra, Risheek Garrepalli, Christos Louizos

The authors evaluate how diffusion language models use context and respond to added mask tokens in single-token tasks. They find accuracy is highest when relevant examples are near the answer position, and extra masks reduce accuracy most for LLaDA, while Dream is more robust. Mask-agnostic fine-tuning reduces, but does not eliminate, this sensitivity; the results concern tested models and tasks.

Key insights

  • MDLMs exhibit a strong locality bias, specifically a "mask-proximate locality" where performance is maximized when relevant information is nearest to the masked question.
  • Extra mask tokens act as significant distractors, leading to an "inverse scaling law" where MDLM performance consistently degrades as the number of appended mask tokens increases.
  • The observed sensitivity to mask tokens is primarily a training artifact, not an architectural limitation, as demonstrated by the effectiveness of the mask-agnostic fine-tuning loss in mitigating this effect.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.