Reproduce “SUNTA: Hierarchical Video Prediction with Surprise-based Chunking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tomoshi Iiyama, Masahiro Suzuki, Yutaka Matsuo
Researchers at The University of Tokyo developed SUNTA, a hierarchical video prediction model that dynamically segments video into temporal chunks based on internal prediction errors or "surprise." This approach enables SUNTA to achieve superior long-horizon video prediction, maintaining accuracy over 250 timesteps where baselines degraded within approximately 10, and accurately identifying ground-truth temporal boundaries with F1 scores up to 0.992.
Key insights
- Prediction errors (surprise) provide a robust and intrinsic signal for dynamically segmenting sequences, leading to chunk boundaries that align better with underlying data dynamics.
- Decoupled, sequential training of hierarchical levels is essential to prevent hierarchical collapse, ensuring that low-level surprise signals remain informative for defining chunk boundaries.
- Top-down surprise, derived from the high-level model's internal consistency checks, enables dynamic chunk detection even when external observations are unavailable during open-loop generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.