Reproduce “Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar, Arnab Kumar Mondal, Zhongyu Jiang, Ian Fasel, Joerg Liebelt

Internalized Visual Thinking (IVT) trains video models to predict future-frame embeddings alongside text answers, then omits image prediction at inference. It evaluates early-event prediction of ongoing actions and next-event prediction of actions not yet begun on three datasets. IVT improves over answer-only fine-tuning across all six settings; Visual CoT remains stronger in two early-event settings. Results concern short-horizon anticipation.

Key insights

  • The benefits of visual foresight for MLLMs can be achieved by learning to predict future *latent representations* during training, without needing explicit, pixel-space generation during inference.
  • Reconstruction-oriented latent representations (e.g., Flux-VAE latents) are most effective as future-state targets, indicating that preserving fine-grained visual structure is crucial for transferring predictive knowledge.
  • Joint training of both textual and predictive objectives (e.g., with a 1:1 data mixture) is critical for effective knowledge transfer and ensures the predictive signal meaningfully influences the model's reasoning capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.