Reproduce “Identifying Interpretable Visual Features in Artificial and Biological Neural Systems” on OpenResearch

About this paper

David Klindt, Sophia Sanborn, Francisco Acosta, Frédéric Poitevin, Nina Miolane

Researchers introduced an automated, quantitative interpretability index (II) validated against human perception, demonstrating that interpretable visual features often emerge from population-level activity in both artificial and biological neural systems. The approach, which employs clustering in activation space, identified features that are more interpretable and stable than individual units across CNNs, macaque IT and AM cortex, and human fMRI data.

Key insights

• Interpretable features in both ANNs and biological systems frequently emerge from the activity of combinations of neurons or voxels, not just single units, supporting the superposition hypothesis.
• The interpretability of these population-level features is empirically linked to properties like disentanglement, universality across subjects, and stability over time.
• Even in untrained models, an interpretability gap exists between individual neurons and activity clusters, with training shifting interpretability from low-level to high-level semantic features.

Summary by alphaXiv.