Reproduce “TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yarui Cao, Kai Liu

Clemson University researchers introduce TLoRA+, a parameter-efficient fine-tuning method for large language models that utilizes a tri-matrix decomposition with a specialized optimizer to assign differential learning rates. This approach consistently surpasses the performance of standard LoRA and TLoRA across various GLUE benchmark tasks and architectures, while maintaining comparable computational efficiency.

Key insights

  • Assigning differential learning rates to the three matrices within the low-rank decomposition (TLoRA+ optimizer) significantly accelerates convergence and improves predictive accuracy, with an optimal ratio range identified between 4.0 and 8.0.
  • Allowing all three matrices (A, B, C) in the tri-matrix decomposition to be trainable enhances the expressive capacity and overall performance compared to fixing two matrices (as in original TLoRA) or using a two-matrix decomposition (LoRA).
  • The TLoRA+ method maintains computational efficiency, with training time per epoch being virtually indistinguishable from standard LoRA, despite a slight increase in trainable parameters.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.