Reproduce “VideoWorld: Exploring Knowledge Learning from Unlabeled Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, Xiaojie Jin
This research investigates whether deep generative models can learn complex knowledge from unlabeled video data alone, contrasting with text-based models. It introduces VideoWorld, an auto-regressive video generation model from Beijing Jiaotong University and ByteDance Seed, enhanced with a Latent Dynamics Model (LDM) to represent multi-step visual changes. The model achieved a 5-dan professional Go playing ability and demonstrated strong performance in robotic manipulation tasks by learning from visual observation without explicit rewards or labels.
Key insights
- Complex knowledge, including rules, reasoning, and planning capabilities, can be acquired by deep generative models solely from raw, unlabeled visual input.
- A compact, multi-step representation of future visual changes (like that provided by LDM) is crucial for efficient and effective knowledge acquisition from videos, overcoming the redundancy of raw pixel data.
- Learning from visual dynamics enables strong generalization across visually distinct environments and complex tasks, indicating potential for "general knowledge learners."
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.