Reproduce “FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning” on OpenResearch

About this paper

Juneyoung Park, Seongbae Lee, Han-Sang Lee, Kyuho Lee, Minjae Kim, Seungheon Hyeon, Kiduk Kwon, Seongwan Kim and 1 more

FoRA introduces a parameter-efficient fine-tuning method that combines Fisher-based layer selection with Stiefel-constrained adapter training, achieving comparable or better performance than LoRA and DoRA while using half the trainable parameters, and recovering adapter effective rank to 88% of its nominal value. The approach yielded up to +5.6 average accuracy gains over vanilla LoRA on LLaMA-3.1-8B and maintained competitive accuracy with AdaLoRA at one-fourth the parameter budget.

Key insights

• Fisher-based layer selection and Stiefel manifold constraints for adapter training are orthogonal components that synergistically boost parameter efficiency and adapter utilization beyond their individual contributions.
• Optimal layer selection for PEFT is highly dependent on the specific LLM architecture, with different families showing distinct patterns of task-informative layers (e.g., early, middle, or bimodal distribution).
• Constraining adapter matrices to the Stiefel manifold effectively prevents "spectral collapse," empirically recovering the adapter's effective rank from approximately 70% to 88-90% of its nominal rank.

Summary by alphaXiv.