Reproduce “See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition” on OpenResearch

About this paper

Chongjie Si, Xiaokang Yang, Wei Shen

A theoretical framework termed Subspace Tuning is proposed to unify and dissect various Parameter-Efficient Fine-Tuning (PEFT) methods from a decomposition perspective. The framework leads to the development of new methods like SSB, which achieves performance comparable to full fine-tuning while using less than 0.1% of parameters, and the Matrix Pattern Constraints (MPC) framework, which enhances existing PEFT methods without additional trainable parameters.

Key insights

• Modifying singular vectors is crucial for effective parameter-efficient fine-tuning, while merely adjusting singular values is insufficient.
• Different decomposition forms in PEFT methods implicitly impose "Matrix Pattern Constraints" that influence performance, with fewer constraints typically leading to better approximation of optimal weights.
• Manipulating both left and right singular vector subspaces simultaneously is more effective for parameter-efficient fine-tuning than adjusting only one side.
• The proposed Matrix Pattern Constraints (MPC) framework can enhance the performance and stability of existing PEFT methods without adding new trainable parameters by alleviating implicit constraints on learned updates.
• Subspace Tuning provides a theoretical lens to categorize and analyze diverse PEFT methods as different forms of subspace manipulation (reconstruction, extension, or combination).

Summary by alphaXiv.