Reproduce “DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xueyu Zhou, Yangrong Hu, Jian Huang
Researchers from The Hong Kong Polytechnic University developed the Dependency-Oriented Sampler (DOS), a training-free decoding strategy for Masked Diffusion Language Models that uses attention weights to guide token selection based on inter-token dependencies. DOS achieves substantial accuracy gains, up to 84.31% on GSM8K, and improves efficiency when integrated with parallel samplers.
Key insights
- Explicitly modeling inter-token dependencies is more critical for high-quality generation in MDLMs than relying solely on token-level uncertainty metrics.
- Attention weights from transformer layers, even from shallow layers, can serve as a robust and training-free signal for capturing syntactic and structural dependencies to guide decoding.
- Combining dependency-aware sampling with existing parallel decoding techniques leads to synergistic benefits, enhancing both output quality and computational efficiency in non-autoregressive generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.