Reproduce “S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baoquan Zhang, Zhehao Yu, Lisai Zhang, Kenghong Lin, Tianran Chen, Yuxi Sun, Yunming Ye, Yao He
S²FT (Sparse Spectrum Fine-Tuning) introduces a parameter-efficient method that transforms observed weight changes into a latent domain exhibiting a truly sparse spectrum, enabling efficient fine-tuning by optimizing only a small fraction of spectral coefficients. It achieves competitive or superior performance across vision and language tasks while utilizing significantly fewer trainable parameters than existing PEFT methods.
Key insights
- The power spectrum of actual weight changes (∆W) in deep neural networks is not sparse but uniformly distributed, which is a fundamental limitation for direct Fourier-based PEFT.
- Inducing spatial smoothness in a weight matrix through row/column rearrangement effectively translates to achieving spectral sparsity, allowing for efficient representation in the Fourier domain.
- A simple, coarse pre-estimation of weight changes is sufficient to effectively guide the spatial rearrangement, minimizing computational overhead for the transformation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.