Reproduce “Two Routes to Scalable Credit Assignment without Weight Symmetry” on OpenResearch

About this paper

Daniel Kunin, Aran Nayebi, Javier Sagastuy-Brena, Surya Ganguli, Jonathan M. Bloom, Daniel L. K. Yamins

Researchers at Stanford University developed a unifying regularization-inspired framework for credit assignment, introducing Information Alignment (IA) as a local learning rule with enhanced metaparameter robustness on ImageNet and two non-local rules, Symmetric Alignment (SA) and Activation Alignment (AA), which match backpropagation's performance and robustness to noisy updates on large-scale tasks.

Key insights

• The instability of Weight Mirror arises from an indefinite matrix in its gradient flow, causing backward weights to diverge without specific stabilization.
• Local learning rules can achieve improved stability and metaparameter robustness through a combination of adaptive optimization, local normalization, and specific regularization primitives like Pnull.
• Non-local learning rules, which rely on weight estimation rather than instantaneous weight transport, can match backpropagation's performance on large-scale tasks while demonstrating robustness to noise.

Summary by alphaXiv.