Reproduce “Similarity of Processing Steps in Vision Model Representations” on OpenResearch

About this paper

Matéo Mahaut, Marco Baroni

This research investigates whether modern vision models converge not only in their final representations but also in their intermediate processing steps, analyzing how diverse architectures and training objectives influence feature hierarchies. The study reveals that while relative layer distances show structural similarity across models, distinct processing strategies emerge regarding the retention of low-level features and the hierarchical organization of semantic information.

Key insights

• Models exhibit shared structural similarity, where layers at similar relative processing depths are most alike, but their absolute processing strategies vary significantly.
• The training objective dictates how models manage feature hierarchies; classification models tend to discard low-level features for abstract semantics, while generative or self-supervised models often retain them throughout.
• Different architectures lead to distinct processing 'smoothness,' with Transformers showing gradual representational changes and CNNs like ConvNeXt demonstrating more erratic or less uniform information flow across layers.

Summary by alphaXiv.