Reproduce “Multimodal foundation models are better simulators of the human brain” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoyu Lu, Qiongyi Zhou, Nanyi Fei, Zhiwu Lu, Mingyu Ding, Jingyuan Wen, Changde Du, Xin Zhao and 3 more

Researchers demonstrated that multimodal foundation models, like their BriVL model, better simulate human brain activity across visual and linguistic tasks compared to unimodal models, particularly in regions involved in multisensory integration. This indicates a closer alignment between multimodal AI representations and biological cognition.

Key insights

  • Multimodal foundation models are significantly more 'brain-like' than unimodal models, especially in high-level visual cortices and regions known for multi-sensory integration (e.g., pSTS).
  • Multimodal pre-training enables the learning of richer, more abstract semantic representations that better align with human brain processing across both visual and linguistic domains.
  • Multimodal foundation models can serve as powerful computational simulators for neuroscientists to investigate multimodal information fusion and functional mapping within the human brain.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.