Reproduce “Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fred Zhangzhi Peng, Alexis Fox, Anru R. Zhang, Alexander Tong
REPR-ALIGN adapts autoregressive (AR) language models to masked diffusion language models (DLMs) by explicitly preserving the AR model's internal representation geometry. This approach demonstrates improved conversion efficiency and higher final performance on code generation tasks compared to existing methods, even with reduced computational and data budgets.
Key insights
- Linguistic representations acquired during AR pretraining are robust and largely generation-order invariant, allowing a decoupling of complex language understanding from specific decoding mechanisms.
- Explicitly preserving the AR model's internal representation geometry during AR→DLM conversion is more effective than solely transferring parameters, leading to substantial gains in adaptation efficiency and quality.
- Not all parameters in the student model require updating during conversion; freezing large blocks (e.g., embeddings and MLP) can significantly increase training throughput while maintaining or improving performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.