Reproduce “ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen
ViBe adds visual feedback to a pretrained humanoid motion tracker by training an attention-based visual extractor and low-rank adapters while keeping the vision encoder and tracker frozen. It learns this connection with policy optimization, without teacher-student distillation. In controlled comparisons, it enables curb and parkour corrections, reaching 95.5% of the privileged-observation policy’s reward on perceptive parkour, a task-specific relative reward.
Key insights
- Perceptive behaviors can be efficiently adapted into existing whole-body controllers post-training by learning only a compact, intermediary connection between pre-trained visual and motor priors.
- Direct policy optimization with visual feedback can approach the performance of policies trained with privileged observations, eliminating the need for teacher-student distillation or auxiliary representation losses.
- The combination of frozen pre-trained vision encoders, a task-conditioned extractor, and LoRA adapters enables robust zero-shot sim-to-real transfer and reactivity to dynamic environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.