Reproduce “Self-similarity Analysis in Deep Neural Networks” on OpenResearch

About this paper

Jingyi Ding, Chengwen Qi, Hongfei Wang, Jianshe Wu, Licheng Jiao, Yuwei Guo, Jian Gao

A universal framework from Xidian University quantifies and regularizes self-similarity in deep neural networks' hidden layer features. This approach improved classification accuracy by 1.2% to 5.3% in MLP and attention-based architectures, linking performance gains to models' inherent self-similar properties.

Key insights

• Visualizing "feature networks" ($G_M$) reveals distinct internal information processing patterns correlating with architectural design, such as global aggregation in MLPs/Transformers versus local processing in CNNs.
• Deep networks typically exhibit a degradation of self-similarity (increased SS rate) during training, reflecting a shift from initial redundancy to optimized, specialized feature representations.
• The effectiveness of self-similarity regularization is tied to a model's intrinsic architectural self-similarity (statistical and geometric invariance), primarily benefiting models like MLPs and attention networks.

Summary by alphaXiv.