Reproduce “Finding trainable sparse networks through Neural Tangent Transfer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianlin Liu, Friedemann Zenke

Neural Tangent Transfer (NTT) introduces a method for identifying trainable sparse neural networks at initialization without requiring labeled data. NTT-initialized sparse networks consistently achieve higher accuracy and faster convergence compared to other foresight pruning methods, particularly for sparsifying convolutional layers.

Key insights

  • The trainability and performance of a sparse network can be predicted by how closely its initial output and Neural Tangent Kernel (NTK) align with those of a dense network in function space.
  • Leveraging NTK theory allows for a principled, label-free approach to characterizing network training dynamics and guiding the search for trainable sparse architectures.
  • Minimizing the discrepancy in both initial function output and the NTK matrix ensures that a sparse network's subsequent supervised training trajectory will closely mimic that of its dense counterpart.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.