Reproduce “World Modeling with Probabilistic Structure Integration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Klemen Kotar, Wanhee Lee, Rahul Venkatesh, Honglin Chen, Daniel Bear, Jared Watrous, Simon Kim, Khai Loong Aw and 8 more
Researchers at Stanford NeuroAI Lab developed Probabilistic Structure Integration (PSI), a self-improving world model that learns from raw video to extract and integrate causal structures like optical flow and depth. PSI achieves state-of-the-art performance in zero-shot optical flow on TAP-DAVIS/Kubric, self-supervised depth on NYUD/BONN, and object manipulation tasks, also improving video prediction by mitigating 'motion collapse'.
Key insights
- Modeling full probability distributions enables crisp, coherent, and multimodal predictions, facilitating precise causal inference from world models.
- Representing visual data as pointer-addressable sequences allows leveraging LLM-style autoregressive transformers and their scaling properties for non-linguistic domains.
- Intermediate causal structures can be extracted zero-shot by perturbing the world model and observing its probabilistic responses, then seamlessly integrated for self-improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.