Reproduce “ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kevin Cai, Chonghua Liu, David M. Chan

University of California, Berkeley researchers introduce Anim-400K, a large-scale dataset of over 425,000 aligned Japanese-English animated video clips, totaling 763 hours, designed to advance automated end-to-end video dubbing and various multimodal AI tasks.

Key insights

  • Large-scale, high-quality, multimodal datasets with true dubbed audio are essential to overcome limitations of pipeline-based methods and advance end-to-end automated dubbing.
  • A "top-down" approach for clip alignment, which recursively merges ASR segments, effectively addresses the challenges of precise temporal synchronization in multimodal dubbing data.
  • Embedding extensive metadata within a dubbing dataset extends its utility far beyond its primary purpose, enabling research in a wide array of multimodal AI tasks such as video summarization and character identification.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.