Reproduce “REPAIR: REnormalizing Permuted Activations for Interpolation Repair” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, Behnam Neyshabur

REPAIR (REnormalizing Permuted Activations for Interpolation Repair) addresses the performance drop observed when linearly interpolating between independently trained deep neural networks. It identifies 'variance collapse' in activation statistics as the cause and corrects it by re-normalizing internal activations, enabling substantial reductions in the interpolation barrier, such as from 16% to 1.5% for ResNet18 on CIFAR-10.

Key insights

  • The primary cause of performance degradation in linearly interpolated deep networks is "variance collapse," where activation variances dramatically decay through deeper layers.
  • The behavior of normalization layers, particularly LayerNorm's test-time normalization, significantly impacts interpolation barriers, explaining previous contradictory findings in the literature.
  • Deep neural networks can have functionally similar solutions connected by low-barrier paths, provided both neuron permutation invariance and activation re-normalization are considered.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.