Reproduce “Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
AJ Piergiovanni, Dahun Kim, Michael S. Ryoo, Isaac Noble, Anelia Angelova
Researchers at Google DeepMind developed a factorized autoregressive decoding approach for online dense video captioning, which generates frequent and detailed descriptions of video segments in real-time without needing future frames. This system outperforms existing state-of-the-art models on benchmarks like ViTT and YouCook2, while achieving similar or better performance with approximately 20% less computational cost.
Key insights
- The factorized autoregressive decoding approach enables linear scaling with video length, reducing computational complexity compared to quadratic scaling in global attention models.
- A single shared text decoder processes multiple video segments independently, promoting parameter efficiency while an autoregressive transformer maintains temporal awareness.
- Cross-segment masking during training allows for efficient parallel computation across segments while preserving the autoregressive property, enhancing training efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.