Reproduce “Conservation Laws for Modern Neural Architectures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc, Nam Nguyen, Tuan Dam, Tan M. Nguyen
Researchers from the National University of Singapore and collaborators developed a unified theoretical framework to characterize conservation laws for modern neural architectures, including SwiGLU FFNs, multihead attention with RoPE, and Mixture-of-Experts. Their work resolves previously open problems by providing complete characterizations and empirically validates these invariants under discrete-time optimization, showing an O(τ^2 * k) error scaling.
Key insights
- Different modern architectural choices fundamentally lead to distinct sets of conservation laws; for example, FFNs with GELU/SiLU exhibit only trivial invariants, while SwiGLU FFNs, multihead attention, and MoE possess specific non-trivial conserved quantities.
- The identified conservation laws represent fundamental geometric constraints within the parameter space, offering insights into the implicit bias of gradient descent for these complex models.
- Theoretically derived conservation laws consistently maintain bounded deviations under practical discrete-time SGD, with an observed error accumulation proportional to O(τ^2 * k), linking continuous-time theory to real-world optimization behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.