Reproduce “Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models” on OpenResearch

About this paper

Aws Khalil, Jaerock Kwon

Researchers at the University of Michigan - Dearborn conducted a zero-shot empirical evaluation of off-the-shelf generative video models to assess their suitability for short-horizon predictive display in vision-based teleoperation systems. The study found that while some models offered higher visual fidelity, none achieved the necessary combination of real-time performance, low prediction error, and stable temporal consistency required for practical latency compensation.

Key insights

• No evaluated off-the-shelf generative video model simultaneously met the demands of low prediction error, stable temporal consistency over short horizons, and real-time inference speed for practical teleoperation.
• Increased model scale (e.g., LTX-13B) did not necessarily improve short-horizon predictive accuracy in a zero-shot context compared to smaller distilled versions (LTX-2B).
• Analysis of per-step temporal error revealed critical inconsistencies (e.g., SVD's unstable predictions) that aggregate pixel-level error metrics alone would not distinguish, highlighting the need for granular temporal analysis.

Summary by alphaXiv.