Reproduce “STaRR: Spatial-Temporal Token-Dynamics-Aware Responsive Remasking for Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinhao Sun, Huaijin Zhao, Maoliang Li, Zihao Zheng, Jiayu Chen, Yun Liang, Xiang Chen
STaRR introduces a training-free remasking framework for Diffusion Language Models that leverages spatial and temporal token confidence dynamics. This approach significantly accelerates inference while maintaining or enhancing generation quality across various reasoning and code generation tasks.
Key insights
- Token confidence during DLM denoising exhibits quantifiable temporal patterns (e.g., stable, sudden surge) and spatial patterns (e.g., consistent, outlier).
- Dynamically adapting remasking based on these spatial-temporal signals with adaptive, token-specific thresholds can significantly improve the speed-quality trade-off.
- Responsiveness optimizations, such as "suspected fast/slow token" mechanisms, are crucial for handling edge cases and ensuring robust, scalable performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.