Reproduce “Seeing Fast and Slow: Learning the Flow of Time in Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner and 1 more

The authors build video models that estimate playback speed, curate slow-motion footage, and generate or reconstruct motion at different temporal scales. Their system uses audio pitch changes to gather training labels, video-based speed prediction, and an explicit speed condition for generation. On 111 verified-rate videos, correlation reaches 0.735; 8× reconstruction obtains FVD 134.3 on a synthetic blurred-input test.

Key insights

  • Audio pitch shifts serve as an effective self-supervised signal for training visual models to detect temporal speed changes in videos.
  • A temporal equivariance-based self-supervised loss, combined with iterative prediction, enables accurate estimation of absolute video playback speed across a wide range, even with limited ground-truth labels.
  • Generating realistic slow-motion videos and performing extreme temporal super-resolution necessitates training on real-world, high-frame-rate data that captures fine-grained motion, unlike artificially slowed standard-FPS content.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.