Reproduce “SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey, Hang Liu
SINKSTATE reduces recurrent-state traffic during linear-attention decoding by maintaining an adaptive low-rank representation. It buffers token updates, applies local SVD updates, and asynchronously consolidates frozen components on the CPU. In evaluation on four Mamba-2/GDN models, Nemotron-3-Nano-4B achieved 1.58× speedup on MATH-500, while peak recurrent-state memory averaged 0.52 of dense baselines and benchmark accuracy changed without a consistent loss.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.