Reproduce “LayerNorm: A key component in parameter-efficient fine-tuning” on OpenResearch

About this paper

Taha ValizadehAslani, Hualou Liang

Research by ValizadehAslani and Liang from Drexel University demonstrates that fine-tuning only LayerNorm parameters in large language models can achieve performance comparable to full fine-tuning while utilizing less than 0.015% of the total model parameters. This approach offers a highly parameter-efficient method for adapting LLMs to downstream tasks, supported by empirical evidence of LayerNorm's significant parameter changes and high Fisher information.

Key insights

• LayerNorm parameters, particularly 'output LayerNorm', consistently exhibit the most substantial changes and highest Fisher information during full fine-tuning, indicating their central role in adapting pre-trained models.
• Layer Normalization's sample-wise normalization is particularly effective for sequential data in language models, allowing it to capture task-specific adaptation patterns more efficiently than batch-dependent normalization methods.
• The critical LayerNorm parameters for adaptation appear largely task-independent, suggesting that universal pre-computed masks could streamline future fine-tuning processes.

Summary by alphaXiv.