Reproduce “M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhe Chen, Heyang Liu, Wenyi Yu, Guangzhi Sun, Hongcheng Liu, Ji Wu, Chao Zhang, Yu Wang and 1 more
Researchers from Shanghai Jiao Tong University, Tsinghua University, and the University of Cambridge introduce M$^3$AV, a large-scale, multimodal, multigenre, and multipurpose audio-visual academic lecture dataset. This dataset features 367 hours of videos from diverse academic fields, coupled with extensive human annotations for speech (including specialized vocabulary) and slide content (with LaTeX-formatted formulae), supporting the development of AI systems for both content recognition and academic knowledge understanding.
Key insights
- Leveraging contextual information, specifically OCR text from slides, significantly improves Automatic Speech Recognition (ASR) performance for rare academic terms.
- Training on real academic lecture speech data enables Text-to-Speech (TTS) models to generate more natural and spontaneous conversational speech.
- Large Multimodal Models (LMMs) show promise in direct visual processing for tasks like Slide-to-Script generation, outperforming cascading OCR-based pipelines in certain metrics, while Retrieval-Augmented Generation (RAG) with external papers can substantially enhance generation quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.