Reproduce “Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng, Mike Zheng Shou
Researchers from the National University of Singapore developed PadCaptioner, a parallelized autoregressive framework for dense video captioning that achieves up to 3.7x total decoding speedup and 3x per-token speedup over prior state-of-the-art models while improving captioning quality. The method leverages a restructured causal dependency graph and a latent global planning mechanism for efficient, high-fidelity event description generation.
Key insights
- Strict sequential token dependency is not universally necessary for temporally distinct events, allowing the causal graph to be restructured for parallel decoding without sacrificing DVC quality.
- Compact latent global event tokens, learned with explicit temporal grounding and adaptive semantic aggregation, effectively summarize event structures and guide subsequent parallel generation.
- An event-factorized attention mechanism is crucial for balancing strong intra-event autoregression with global inter-event awareness, preventing interference between parallel event subchains and restoring performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.