Reproduce “JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kehan Wen, Ziming Li, Siyuan Luo, Fan Shi
JEPA-x trains visual and privileged physical trajectories together, using a shared action-conditioned predictor to match each history’s predictions to both paired future representations. At deployment, only vision is used. In simulated tabletop tasks, it reaches 78.2% ± 1.3 success, versus 53.6% ± 2.5, and lower rollout drift (0.104 ± 0.003 versus 0.361 ± 0.024); these results concern this suite and do not guarantee better control in every task.
Key insights
- Explicitly constraining the action-conditioned *evolution* of visual latent representations with corresponding privileged physical trajectories, rather than just their content, leads to more forecastable dynamics.
- There is a critical distinction: improving the *decodability* of physical variables from a latent state does not automatically enhance the *forecastability* of its latent dynamics or its overall *control utility*.
- Cross-modal predictive coupling across the evolution of visual and physical latents is more critical for improving forecastability than simply sharing a predictor or using frame-level alignment, and correct trajectory pairing is essential.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.