Reproduce “Register Tokens for Bounded-State Reasoning in Diffusion Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Albert Ge, Chandan Singh, Yufan Zhuang, Xiaodong Liu, Jianfeng Gao, Frederic Sala

The paper introduces continuous register states that let diffusion language models carry progress across fixed-size generation windows. After denoising a window, the model stores hidden states in fixed positions, clears generated text, and continues with the prompt and registers. On Dream-7B, registers reach 40.9% pass@1 on MBPP, versus 21.4% with discrete-text carry. This result is bounded to the tested code setting; full-sequence SFT performs better on all eight main math rows.

Key insights

  • Register states are causally useful across chunk boundaries: replacing them with noise reduces GSM8K accuracy from 46.4% to 32.7%, while resetting them reduces it to 21.5%.
  • The state encodes task-relevant information, including the final answer, running total, answer sign, and next operation; probes achieve correlations of 0.85 and 0.84 for the first two variables and 94.1% and 80.0% accuracy for the latter classifications.
  • Registers are most valuable when outputs genuinely exceed the active window. Full-context or full-sequence training can remain better for shorter mathematical generations, while code and long-horizon arithmetic show clearer benefits from persistent state.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.