Reproduce “VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yongchao Huang
VJEPA predicts probability distributions over future latent representations, extending JEPA’s point predictions without reconstructing pixels. Its predictor is trained against target-encoder representations, with a regularization term toward a prior. In a synthetic noisy linear system, VJEPA recovers the signal at scale 8 (−2.2 dB) with test R²=0.870, below deterministic JEPA’s 0.930; this is not a planning trial.
Key insights
- VJEPA's objective maximizes a variational lower bound on mutual information between past and future representations, causing it to act as a Predictive Information Bottleneck and prioritize predictable, task-relevant information.
- Unlike generative models that act like nonlinear PCA and encode high-variance nuisance, VJEPA functions as a nonlinear Canonical Correlation Analysis (CCA), inherently filtering out nuisance variability by focusing on shared predictable information.
- BJEPA's Product of Experts formulation enables modular injection of task-specific priors, facilitating zero-shot task transfer and robust planning through latent Bayesian filtering by fusing learned dynamics with explicit constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.