Reproduce “Real-time Online Video Detection with Temporal Smoothing Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yue Zhao, Philipp Krähenbühl

Temporal Smoothing Transformers (TeSTra) from the University of Texas at Austin introduces a kernel-based attention mechanism that enables real-time online video detection and anticipation with constant computational cost per frame. The model achieves state-of-the-art results on THUMOS’14 with 71.2% mAP for online detection and 17.6% action recall on EPIC-Kitchen-100, while running 6 times faster than comparable sliding-window Transformers for long histories.

Key insights

  • The attention mechanism can be reformulated as a kernel function, allowing specific kernel choices (like the Laplace kernel) to enable recursive, constant-time updates essential for streaming.
  • Separating temporal and feature kernels, where the temporal kernel allows for constant-time updates, is effective for efficient long-term memory processing in Transformers.
  • Data augmentation like MixClip, which diversifies long-term contextual cues, significantly improves model generalization and anticipation performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.