Reproduce “Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Utkarsh A. Mishra, Yongxin Chen, Danfei Xu, Yang Liu, Xi Chen, Jiayuan Mao
Georgia Tech and Amazon FAR researchers characterized the "video-action generalization (VAG) gap," where finetuning robot policies on action data diminishes the compositional priors of underlying video foundation models. They developed Temporal Ratio (TR), an attention-based metric to quantify reliance on future predictions, and designed a TR-adaptive guidance method to enhance out-of-distribution compositional task success, improving it to 59.4% in simulation and 83.3% in real-world bimanual tasks.
Key insights
- The video-action generalization (VAG) gap is a fragmented issue across VAM design choices; low-rank adaptation (LoRA) and joint training with the video backbone yield better compositional generalization than full finetuning.
- Temporal Ratio (TR) serves as an effective diagnostic tool, predicting a model's compositional generalization capacity and dynamically reflecting task phases (rising during planning, dropping during precise manipulation).
- The action head's effective utilization of physically plausible video predictions is crucial; simply generating a correct future rollout is insufficient if the policy does not leverage it for action generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.