Reproduce “MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sten Rüdiger, Sebastian Raschka

Researchers at the RAIR Lab and an independent researcher introduced Minor Component Adaptation (MiCA), a parameter-efficient fine-tuning method for large language models that adapts underutilized minor singular directions of pre-trained weights. MiCA achieved up to 5.9x better knowledge acquisition on novel domain data while using 6-60% fewer trainable parameters than LoRA, without substantially degrading general capabilities.

Key insights

  • Minor singular directions of pre-trained weight matrices offer a plastic and effective subspace for integrating new, task-specific knowledge, as opposed to dominant components saturated with generic knowledge.
  • Constraining adaptation to these spectrally identified minor components enhances knowledge acquisition and significantly reduces the number of trainable parameters.
  • Explicitly selecting adaptation subspaces based on spectral properties of pre-trained weights, rather than implicitly learning them, can lead to more stable and efficient model specialization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.