Reproduce “Self-supervised models of audio effectively explain human cortical responses to speech” on OpenResearch

About this paper

Aditya R. Vaidya, Shailee Jain, Alexander G. Huth

Self-supervised learning models of speech effectively predict human cortical responses to natural speech, demonstrating superior performance over traditional and supervised baselines. Their internal representational hierarchies closely align with the brain's known processing stages, capturing acoustic, phonetic, and semantic information at different depths.

Key insights

• Self-supervised learning models provide a new state-of-the-art in computational modeling of human cortical responses to natural speech, significantly outperforming prior methods.
• A strong hierarchical correspondence exists between the processing depth of SSL model layers and the functional organization of human auditory and language cortices.
• The internal representations of SSL models capture a multi-faceted hierarchy of information, evolving from low-level acoustic features to high-level phonetic and semantic content across their depth.

Summary by alphaXiv.