Reproduce “The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Oswaldo Ludwig
Oswaldo Ludwig introduces KappaTune, a method that uses the condition number of weight tensors as a scale-invariant proxy for information encoding to selectively fine-tune pre-trained neural networks. The approach mitigates catastrophic forgetting, reducing perplexity change on WikiText-2 by over 6x compared to LoRA, and improves generalization in LLMs without requiring access to original training data.
Key insights
- A high condition number in a well-trained model signifies a specialized unit that has efficiently encoded discriminative knowledge by selectively amplifying certain input directions.
- Units with low condition numbers (approaching 1) maximize differential entropy for a fixed weight norm, implying a more general, adaptable representational capacity suitable for new knowledge acquisition without disrupting specialized information.
- Selectively adapting these low-kappa tensors allows models to learn new tasks or modalities while significantly preserving pre-trained knowledge, offering a principled, data-agnostic approach to continual learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.