Reproduce “T2-z: Test-Time Optimization of Environment Latents for Robotic World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenyuan Zhou, Yining Hong, Huang Huang, Yunfan Jiang, Wenlong Huang, David Held, Vitor Campagnolo Guizilini, Yue Wang and 4 more
T2-Z adapts an action-conditioned world model to hidden environmental variation by optimizing a compact latent from observed interaction trajectories, without labels for hidden factors. The trained model stays frozen during test-time adaptation. Across four real-world tasks, average tracked-state prediction error fell from 19.30 to 10.85 pixels and action success rose from 26.47% to 70.80%; these are task averages, not guarantees.
Key insights
- A small number of interaction trajectories can provide enough evidence to adapt predictions in the evaluated tasks; performance generally improves up to 40 latent-optimization steps and then largely levels off.
- Prediction-driven latents show structure associated with varying physical factors and are less dominated by background appearance in the tested analysis, though this does not establish unique or causal encoding.
- Better visual similarity is not always the same as better physical prediction: task-relevant errors and action success can improve even when a perceptual metric worsens on an individual task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.