Reproduce “Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anxing Xiao, Nuwan Janaka, Tianrun Hu, Anshul Gupta, Kaixin Li, Cunjun Yu, David Hsu

Robi Butler lets a remote user direct a household robot by speaking and pointing to an object in its first-person video. An LLM aligns the language and gesture, then organizes the request into robot skills. In ten tasks repeated three times by one expert, task success was 96.7% and planning success 100%; the demonstration covers selected tasks, not arbitrary chores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.