Reproduce “Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment” on OpenResearch

About this paper

Soh Takahashi, Masaru Sasaki, Ken Takeda, Masafumi Oizumi

This study from the University of Tokyo utilized unsupervised alignment with Gromov-Wasserstein Optimal Transport to analyze how deep neural networks map to human object similarity judgments at both fine- and coarse-grained levels. It demonstrated that models incorporating linguistic information achieve strong fine-grained alignment, whereas self-supervised models primarily develop coarse categorical structures, suggesting distinct roles for different learning paradigms.

Key insights

• Linguistic information, whether explicit (supervised labels) or integrated through contrastive learning (CLIP), is crucial for DNNs to develop fine-grained object representations that align with human judgments.
• Self-supervised learning, particularly contrastive methods, can robustly form coarse human-like category structures even without linguistic input, potentially modeling prelinguistic cognitive development in humans.
• Unsupervised alignment methods like GWOT offer a more discriminative approach than conventional RSA, allowing for a nuanced assessment of fine- and coarse-grained structural correspondences between different representational systems.

Summary by alphaXiv.