Reproduce “On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men and 28 more

Qwen3.8-Flash-Next combines Gated DeltaNet, Qwen Sparse Attention, Gated Residuals, and off-accelerator n-gram capacity to process long contexts with recurrent memory and selective block retrieval. At one million tokens, its sparse attention kernel is 7.6× faster for prefill and 4.9× for decode than dense attention under reported settings. These are kernel measurements, not end-to-end serving speedups; the base model beats Qwen3.7-Plus on 8 of 14 benchmarks.

Key insights

  • A holistic design approach, considering loss, downstream performance, training/inference efficiency, and training stability simultaneously, is crucial for developing highly effective large language models.
  • The Gated Residual architecture, particularly its GatedNorm component, significantly enhances training stability by reducing the frequency and magnitude of gradient-norm spikes and activation outliers.
  • Optimal hyperparameters for large-scale training are highly specific to the architectural design and optimizer used, with batch-size warmup found to be unnecessary under the new recipe.
  • Adding n-gram embedding layers can efficiently scale model capacity, but the optimal vocabulary size for pre-training loss may diverge from that for downstream task performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.