Reproduce “NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Snehesh Shrestha, Yantian Zha, Saketh Banagiri, Ge Gao, Yiannis Aloimonos, Cornelia Fermüller
Researchers at the University of Maryland, College Park, introduce NatSGLD, a multimodal dataset for human-robot interaction that integrates speech, gestures, Linear Temporal Logic (LTL) task annotations, and expert robot demonstrations. This dataset supports robots in understanding and executing complex, multi-step tasks in a simulated kitchen environment, demonstrating that combining speech and gestures improves the accuracy of LTL task interpretation.
Key insights
- Integrating speech, gestures, formal logic (LTL), and robot demonstrations provides a more complete supervision signal for robot learning, addressing both high-level task understanding and low-level execution.
- Natural human communication for complex tasks critically relies on the synergistic use of both speech and gestures, where gestures provide essential tacit, context-dependent details complementing verbal instructions.
- Formal representations such as LTL are crucial for enabling robots to derive robust plans and reason about the complex temporal and sequential properties embedded in human commands.
- Wizard-of-Oz experimental designs effectively elicit natural human-robot interaction data by fostering the illusion of a fully autonomous and capable robot, ensuring spontaneous and unconstrained human input.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.