Reproduce “Low-Rank Interconnected Adaptation across Layers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yibo Zhong, Jinman Zhao, Yao Zhou

Researchers from Sichuan University and the University of Toronto developed Lily, a parameter-efficient fine-tuning framework that overcomes the low-rank update limitation of LoRA by employing an interconnected adapter structure and data-dependent routing. Lily achieves higher-rank weight updates and superior performance across various modalities and architectures, often surpassing full fine-tuning on NLU tasks and requiring significantly fewer parameters than previous methods.

Key insights

  • Decoupling A and B adapters with hierarchical sharing and data-dependent routing effectively overcomes the low-rank update constraint of conventional LoRA, enabling more expressive and higher-rank adaptations.
  • Data-dependent routing is crucial for preventing B experts from converging to identical behaviors, ensuring diversity and allowing dynamic, comprehensive adaptation across layers.
  • Increasing the *rank* of adapters generally leads to greater performance gains than simply increasing the *number* of experts for a fixed parameter budget, suggesting a nuanced trade-off in parameter allocation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.