Reproduce “Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks” on OpenResearch

About this paper

Jialin Zhao, Yingtao Zhang, Xinghang Li, Huaping Liu, Carlo Vittorio Cannistraci

Researchers at Tsinghua University developed Sparse Spectral Training (SST), a memory-efficient pre-training framework that closely approximates full-rank learning dynamics by updating all singular values and selectively updating singular vectors. This method significantly reduces the performance gap to full-rank training by up to 97.4% in language modeling perplexity and 66.7% in machine translation BLEU scores, while maintaining memory consumption comparable to other parameter-efficient methods.

Key insights

• Continuously updating all singular values while selectively updating singular vectors based on their magnitudes allows SST to closely mimic full-rank training dynamics and retain comprehensive expressive power.
• SVD-based initialization and periodic re-orthogonalization are crucial for maintaining learning stability and preventing gradient stagnation (saddle points) that can occur in iterative low-rank adaptation techniques.
• The sparse spectral approach is effective across diverse architectures and tasks, including hyperbolic neural networks, where it helps mitigate overfitting by judiciously constraining the parameter search space.

Summary by alphaXiv.