Reproduce “Layer-wise Linear Mode Connectivity” on OpenResearch

About this paper

Linara Adilova, Maksym Andriushchenko, Michael Kamp, Asja Fischer, Martin Jaggi

This research introduces Layer-wise Linear Mode Connectivity (LLMC), demonstrating that individual layers of deep neural networks can be linearly interpolated without loss barriers, even when the full network exhibits such barriers. It reveals that the overall averaging barrier primarily arises from the network's middle layers and correlates this behavior with model robustness properties.

Key insights

• Individual layers within deep neural networks generally exhibit linear mode connectivity, meaning they can be interpolated without significant loss barriers, a finding robust across various models and training environments.
• The overall linear averaging barrier in deep networks tends to emerge and become most pronounced when the middle layers of the network are included in the averaging process, indicating their critical role in connectivity.
• The presence of layer-wise linear mode connectivity is linked to model robustness and flatness, with models trained to achieve flatter minima generally showing fewer and shallower layer-wise barriers.

Summary by alphaXiv.