Reproduce “SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zian Liu, Yiwen Hu, Zican Dong, Tian Xie, Wayne Xin Zhao, Yucheng Ding, Ran Tao, Bryan Dai
SpectralShift adapts Gated DeltaNet for longer-context retrieval by reshaping the recurrent state’s retention pattern and scaling the alpha-projection learning rate during continued pretraining. In a 1.5B-A0.6B GDN-MoE model extended to 128K, its mean RULER score across five evaluation lengths was 55.18, versus 52.38 for continued pretraining. The result is limited to tested smaller-scale settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.