Reproduce “Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amine Elhafsi, Daniel Morton, Marco Pavone
Stanford University and NVIDIA Research introduced the SMS framework, which enables robots to plan actions by systematically combining advanced 3D scene reconstruction via 3D Gaussian Splatting, semantic understanding from foundation models, and physics simulation. This approach allows robots to reason about physical interactions and generalize across diverse tasks, demonstrating superior performance in billiards-inspired manipulation and quadrotor landing scenarios.
Key insights
- The SMS framework provides a systematic integration of 3D scene reconstruction, semantic understanding from foundation models, and physics simulation for robot action planning.
- Material properties can be inferred from visual appearance using vision-language models, eliminating the need for manual specification in physics models.
- Physics-informed planning allows robots to reason about complex physical interactions and generalize to novel environments beyond training distributions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.