Reproduce “Improving Semantic Understanding in Speech Language Models via Brain-tuning” on OpenResearch

About this paper

Omer Moussa, Dietrich Klakow, Mariya Toneva

Researchers from Max Planck Institute for Software Systems and Saarland University developed "brain-tuning," a method to fine-tune speech language models with fMRI recordings, which enhanced brain alignment with semantic regions by approximately 30% for self-supervised models and improved performance across various semantic downstream tasks. The approach also reduced models' reliance on low-level speech features for brain alignment.

Key insights

• Directly integrating human fMRI signals into speech model training can induce brain-relevant semantic bias, enhancing their understanding.
• Brain-tuning reduces the reliance of speech models on low-level speech features for alignment with semantic brain regions, indicating a shift towards more abstract semantic processing.
• A relatively small amount of fMRI data (less than 0.7% of original training size) can efficiently improve speech model capabilities, and smaller brain-tuned models can achieve alignment comparable to larger pretrained ones.

Summary by alphaXiv.