Reproduce “PELA: Learning Parameter-Efficient Models with Low-Rank Approximation” on OpenResearch

About this paper

Yangyang Guo, Guangzhi Wang, Mohan Kankanhalli

PELA introduces a method to compress large pre-trained models into genuinely smaller, low-rank versions via an intermediate pre-training stage, enabling their direct use for fine-tuning and inference. This approach maintains performance close to original models (typically within 0.6-1.0% accuracy) while reducing parameters and FLOPs by 33% to 66% across various vision and vision-language tasks.

Key insights

• Direct low-rank approximation leads to significant performance degradation, requiring a dedicated intermediate pre-training stage for effective capability recovery.
• Combining feature distillation for aligning intermediate representations with regularized weight perturbation to constrain reconstruction error is crucial for maintaining performance in low-rank compressed models.
• Creating a genuinely smaller, standalone pre-trained model frees up computational resources, allowing for optimizations like larger downstream training batch sizes, which can further enhance performance beyond the original model.

Summary by alphaXiv.