Reproduce “Large Flow Language Models” on OpenResearch

About this paper

Arnav Chavan, Chanhyuk Lee, Udbhav Bamba, Jaehoon Yoo, Seunghoon Hong, Aditi Raghunathan, Jinwoo Kim

Lafa adapts autoregressive language models into continuous-flow models that draft token-embedding blocks together, then verifies drafts against its retained next-token predictor. Self-speculative decoding commits the longest prefix matching greedy predictions. On one H100 GPU at batch size one, Lafa 7B with block size 16 reached 100.2 tokens per second and 74.5% average accuracy, versus 39.5 and 73.9% for AR SFT across three tasks.

Key insights

• Flow-based block training can complement, rather than replace, autoregressive prediction: Lafa outperforms matched autoregressive fine-tuning baselines on six-benchmark average accuracy at 0.5B, 7B, and 8B scales.
• Within-block parallelism benefits from bidirectional attention and learned asynchronous schedules; ablations also favor norm-constrained embedding updates and a combination of denoising losses.
• Drafting with flows and verifying autoregressively provides a practical speed-quality tradeoff, though larger blocks tend to increase throughput at some cost to accuracy.

Summary by alphaXiv.