Reproduce “SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunnan Wang, Kecheng Zheng, Jianyuan Wang, Minghao Chen, David Novotny, Christian Rupprecht, Yinghao Xu, Xing Zhu and 3 more

Researchers introduce SceneScribe-1M, a dataset featuring one million "in-the-wild" videos, providing over 4,000 hours of content annotated with detailed textual descriptions, precise camera parameters, dense depth maps, motion masks, and consistent 3D point tracks. This resource is designed to support both 3D geometric perception and video synthesis tasks, aiming to facilitate the development of World Foundation Models.

Key insights

  • Large multimodal language models can be effectively deployed for automated quality filtering and comprehensive semantic annotation of video content, ensuring high-quality and detailed descriptions.
  • Integrating advanced visual SLAM (MegaSaM) with mono-depth priors and a bundle adjustment layer can robustly estimate precise camera parameters and dense depth maps even in unconstrained dynamic video scenes.
  • A strategic multi-view sampling approach can successfully isolate camera motion diversity from object motion intensity, enabling the creation of specialized dataset subsets tailored for specific tasks without compromising other important characteristics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.