Reproduce “KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides, Danilo Mandic
KromHC introduces a method for Manifold-Constrained Hyper-Connections that leverages Kronecker products of small, exactly doubly stochastic matrices, resolving the trade-off between numerical stability and parameter efficiency in deep neural networks. This approach achieves an O(n²C) parameter complexity, a reduction from O(n³C) or O(nC · n!), while improving LLM pretraining performance with higher CORE scores and lower gradient norms.
Key insights
- Leveraging the Kronecker product property that the product of doubly stochastic matrices is also doubly stochastic allows exact constraint enforcement for large matrices from smaller, manageable components.
- Tensorizing the residual stream and applying mixing along individual modes with small, exactly doubly stochastic factor matrices drastically reduces parameter complexity to O(n²C) from higher orders.
- The Birkhoff-von-Neumann theorem provides a practical way to guarantee exact double stochasticity for small factor matrices by reparameterizing them as convex combinations of permutation matrices.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.