Reproduce “Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals” on OpenResearch

About this paper

Simo Ryu, Seunghyun Seo, Jaejun Yoo

Efficient Residual Encoding (ERE) compresses fully fine-tuned models by leveraging the low-rank structure of weight residuals, the differences between fine-tuned and pre-trained weights. This post-hoc method drastically reduces storage requirements (e.g., to 6% for NLU models and less than 1% for image generation models) while largely maintaining or occasionally enhancing performance across various modalities.

Key insights

• Weight residuals (∆θ) exhibit significantly stronger low-rank characteristics compared to the full model weights, indicating high compressibility.
• The effective ranks of weight residuals vary dynamically and unpredictably across different layers within a model, necessitating a non-uniform, layer-wise rank allocation strategy for optimal compression.
• Weight residuals are more robust to random noise and low-rank approximation than full weights, implying that aggressive compression of these residuals can be achieved with minimal impact on model performance.

Summary by alphaXiv.