Reproduce “Musec: MomentUm SpEctral Clipping for Stable Muon-type Training” on OpenResearch

About this paper

Zhuanghua Liu, Menglian Wang, Luo Luo

MUSEC modifies Muon-style optimization by clipping, rather than flattening, the singular values of momentum matrices. Its SVD-free Soft Musec implementation remains stable at learning rates where Muon diverges, controls spectral growth across attention and MLP weights, adds roughly 2–3% step-time overhead, and achieves comparable or slightly lower validation loss than competing clipping methods on models up to 1.63 billion parameters.

Key insights

• Preserving small and moderate singular values avoids the update amplification caused by forcing every active direction toward unit magnitude, while capping dominant directions limits unstable spectral growth.
• Clipping the momentum state itself provides broader regulation than methods targeting only query-key weights or clipping an update without modifying the recurrence.
• The experiments support a link between optimizer spectral behavior and training stability: Soft Musec and SPECTRA keep spectral norms below about 10, whereas Muon reaches roughly 300–400 in several matrix groups.

Summary by alphaXiv.