Reproduce “Next Block Prediction: Video Generation via Semi-Autoregressive Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuhuai Ren, Shuming Ma, Xu Sun, Furu Wei
A semi-autoregressive framework, Next-Block Prediction (NBP), is presented for video generation, enabling more efficient and higher-quality outputs by predicting video content in blocks instead of token by token. The framework achieves an 11-fold inference speedup and consistently lower Fréchet Video Distance scores compared to traditional Next-Token Prediction models, particularly when using a row-wise block generation strategy.
Key insights
- Redefining the generation unit from individual tokens to 'blocks' significantly improves inference efficiency by enabling parallel prediction, thus reducing the total number of sequential forward passes required.
- Bidirectional attention applied within blocks, while preserving causality between blocks, effectively models 2D spatial correlations within video frames, leading to enhanced generation quality.
- The optimal block shape is crucial, with row-wise prediction (1x1x16 block) providing the best balance between speed and quality, as it simplifies the modeling of token relationships compared to larger, more complex block configurations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.