Reproduce “Representation-based Masked Diffusion Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yangrong Hu, Ding Huang, Xueyu Zhou, Jian Huang
RMDM coordinates parallel masked-token generation by conditioning its denoiser on a shared latent sampled from a Gaussian prior. It learns this latent by aligning pretrained encoder representations with MeanFlow, then reuses one latent throughout denoising. On OpenWebText at 16 steps, it reaches 126.58 versus 213.44 GenPPL and 1.87 versus 1.73 judge score against retrained MDLM. Results cover unconditional generation at roughly GPT-2 scale, with unequal pretrained representation access.
Key insights
- The method frames parallel-generation errors through conditional total correlation: a useful global latent can reduce the dependence remaining among masked tokens and make factorized sampling a better approximation to joint sampling.
- The data-dependent representation is essential; replacing it with an independent Gaussian vector substantially worsens generation perplexity, showing that the gain is not merely due to an additional random conditioning channel.
- Global conditioning is most valuable under aggressive sampling: RMDM’s advantage over MDLM grows as the denoising budget decreases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.