Reproduce “Position: Curvature Matrices Should Be Democratized via Linear Operators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Felix Dangel, Runa Eschenhagen, Weronika Ormaniec, Andres Fernandez, Lukas Tatzel, Agustinus Kristiadi
The `curvlinops` library introduces a unified linear operator interface for various curvature matrices in deep learning, encapsulating computational complexities to simplify their application in tasks like optimization and analysis. This framework integrates with advanced numerical linear algebra methods, including Randomized Linear Algebra, demonstrating scalability on large neural network architectures.
Key insights
- A generalized linear operator abstraction can effectively unify the representation and computation of diverse curvature matrices, from exact Hessians to various approximations like GGN and KFAC.
- Encapsulating low-level numerical and automatic differentiation complexities within a consistent interface significantly simplifies the practical application of curvature information in deep learning tasks.
- This abstraction fosters extensibility and interoperability, enabling seamless integration with advanced numerical linear algebra libraries (e.g., SciPy) and facilitating the adoption of Randomized Linear Algebra methods for curvature analysis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.