Reproduce “MindShot: Multi-Shot Video Reconstruction from fMRI with LLM Decoding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenwen Zeng, Yonghuang Wu, Yifan Chen, Xuan Xie, Chengqian Zhao, Feiyu Yin, Guoqing Wu, Jinhua Yu

MindShot, a framework developed at Fudan University, reconstructs multi-shot videos from fMRI signals by segmenting brain activity, using a Large Language Model to generate semantic keyframe captions, and synthesizing a dedicated dataset. This approach accurately decodes complex visual narratives from brain activity, moving beyond single-shot limitations.

Key insights

  • Explicitly segmenting mixed fMRI signals into shot-specific embeddings is crucial for disentangling neural activities and achieving precise semantic reconstruction of individual shots.
  • Leveraging LLMs to decode high-level keyframe captions from fMRI effectively bypasses the fMRI's low temporal resolution, providing robust semantic prompts for subsequent video generation.
  • A multi-task loss function combining caption generation, contrastive alignment, and noise prediction loss synergistically enhances semantic representation extracted from fMRI data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.