Reproduce “Retrofitting Linear Attention into Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinha Kim, Younghun Roh, Jaeyeon Kim
Researchers developed a block-hybrid attention mechanism and a two-stage retrofitting procedure for Diffusion Language Models (dLLMs), enabling up to 1.73x faster inference throughput and 1.3x higher concurrency on a single H200 GPU while largely preserving generation quality. This method integrates linear attention for past context and softmax attention for the active block, addressing efficiency bottlenecks in dLLM inference.
Key insights
- A hybrid attention approach effectively balances the need for fine-grained bidirectional attention within active blocks with the efficiency of linear attention for summarizing long historical contexts in dLLMs.
- Architectural modifications can be cost-effectively applied to large pretrained models through a targeted two-stage post-training process, significantly reducing the computational burden compared to training from scratch.
- Leveraging linear attention's fixed-size recurrent state fundamentally decouples attention cost and memory usage from sequence length, yielding constant per-token attention cost for previously committed tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.