Reproduce “KLASS: KL-Guided Fast Inference in Masked Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seo Hyun Kim, Sunwoo Hong, Hojung Jung, Youngrok Park, Se-Young Yun

KLASS, a training-free adaptive sampling method for masked diffusion models, leverages Kullback-Leibler divergence and confidence scores to identify and unmask multiple "stable" tokens in parallel. This approach achieved up to 2.78x faster inference while enhancing generation quality, including a 4.8 percentage point accuracy improvement on MATH reasoning and superior FID/IS scores for image generation.

Key insights

  • Temporal stability, quantified by KL divergence between successive predicted token distributions, is a reliable signal: correct predictions consistently exhibit lower KL scores than incorrect ones.
  • Combining both confidence and KL stability criteria is crucial for optimal performance, achieving a superior balance of speed and accuracy compared to using either criterion in isolation.
  • A theoretical proposition suggests that dynamically unstable tokens are often incorrect, providing a justification for delaying unmasking based on KL divergence to improve generation quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.