Reproduce “RoboDreamer: Anticipatory Humanoid Locomotion with Predictive State-Space Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhe Li, Yangyang Wei, Xichen Yuan, Zhenzhe Zhang, Weihao Yuan, Shanghang Zhang, Jianfei Yang
RoboDreamer trains a humanoid motion tracker to act through missing recent observations. It first trains a teacher on clean observations, then distills a student exposed to learned mask tokens and next-observation prediction. Using Mamba as its temporal backbone, it shows lower tracking error than MLP and Transformer in a matched action-chunking test. Real-robot outage robustness was not systematically tested.
Key insights
- Robustness to missing sensing is learned explicitly through structured masking rather than treated only as generic observation noise; the teacher provides stable supervision while the student reconstructs the hidden state.
- Next-observation prediction improves the latent representation’s connection to local dynamics and becomes especially useful as the controller predicts farther ahead with larger action chunks.
- Mamba provided a favorable accuracy–latency trade-off: on FineDance it reached 146.72 mm keybody error with 4.4 ms latency, compared with 149.79 mm and 9.8 ms for the tested Transformer and 163.35 mm and 3.5 ms for the MLP.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.