Reproduce “Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tejas Pradeep Shirodkar

DDC, the Dead-Direction Conditioner, separates neural-network updates into motion along function-preserving symmetry orbits and motion across them, conditioning the two components separately and projecting descent onto the quotient. The authors test it on an over-trained depth-12 language model. In the language-model experiment, 32 of 65 layer-by-observable cells are rate-readable, versus 7 under AdamW; this is a gated result in one setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.