Reproduce “Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks” on OpenResearch

About this paper

Ying Chen, Aoxi Li, Jihun Kim, Javad Lavaei

The work by Chen et al. reveals that purely low-rank neural networks exhibit "orthogonal blindness," limiting their ability for general function approximation, and introduces Diagonal plus Low-Rank (DLoR) networks, which restore universal approximation properties with parameter-efficient structures.

Key insights

• Purely rank-1 hidden layers can perfectly interpolate scalar datasets but suffer from "orthogonal blindness," leading to a fundamental collapse in capacity for multi-dimensional function approximation.
• The minimal addition of a sparse diagonal component (αI) to a low-rank matrix is sufficient to overcome orthogonal blindness, thereby restoring the topological capacity for general function approximation without a dense prior.
• Universal Approximation Theorem (UAT) can be fully restored in DLoR networks for general, non-affine continuous activation functions, and existing UAT guarantees can be systematically transferred from dense to DLoR architectures.

Summary by alphaXiv.