Reproduce “Robot Operation of Home Appliances by Reading User Manuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jian Zhang, Hanbo Zhang, Anxing Xiao, David Hsu
ApBot, a robot system developed at the National University of Singapore, operates novel household appliances by interpreting user manuals and visual observations, demonstrating zero-shot generalization capabilities. The system significantly outperforms direct large vision-language model (LVLM) control policies in simulation, achieving higher task success rates and greater robustness across diverse appliance types.
Key insights
- Structured internal representations, derived from unstructured text and grounded in the physical world, are critical for robust robot decision-making in complex, long-horizon tasks.
- Combining LVLM reasoning with code-based policies for low-level action sequences enhances precision and correctness, especially for actions with large value ranges or complex transitions.
- Closed-loop model calibration, enabled by visual feedback and state re-estimation, is essential for compensating for initial model inaccuracies or ambiguities in user manuals, significantly improving task robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.