Reproduce “Look Before You Leap: Using Serialized State Machine for Language Conditioned Robotic Manipulation” on OpenResearch

About this paper

Tong Mu, Yihao Liu, Mehran Armand

Researchers at Johns Hopkins University and the University of Arkansas developed a state-aware imitation learning framework that uses Large Language Models (LLMs) to generate and filter Finite State Machines (FSMs) for robust robotic demonstration collection. This method achieved up to 98.0% success rates on complex long-horizon tasks like Towers of Hanoi, significantly outperforming control methods that struggled to generalize with random environment initializations.

Key insights

• Leveraging LLMs to *indirectly* generate and filter formal FSM definitions ensures logical consistency and correctness in task planning, overcoming challenges of direct, unstructured LLM plan generation.
• Systematic, state-aware demonstration collection, covering all valid FSM transitions, is crucial for training robust policies for long-horizon tasks that involve dynamically evolving environments.
• Decoupling low-level object instantiation from high-level task logic, while preserving diversity through constrained initial state randomization, significantly improves generalization in complex manipulation scenarios.

Summary by alphaXiv.