Reproduce “Optimizing Mode Connectivity via Neuron Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

N. Joseph Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, Rongjie Lai

Researchers from Rensselaer Polytechnic Institute and IBM Research optimized neural network mode connectivity by explicitly accounting for weight permutation symmetry via neuron alignment. This approach substantially reduces the robust loss barrier for adversarially robust models, identifying models along the path with improved clean and robust accuracy.

Key insights

  • Weight permutation symmetry plays a critical role in the connectivity of neural network loss landscapes, with symmetric alignment revealing flatter, lower-loss paths.
  • Neuron alignment facilitates smoother interpolation of learned features, promoting similarity in intermediate activations along the connected path.
  • The previously observed robust loss barrier between adversarially robust models is largely an artifact of not accounting for weight permutation symmetry.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.