Reproduce “Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes
Swim2Real, an automated pipeline, calibrates a 16-parameter robotic fish simulator using Vision-Language Model (VLM) feedback derived from swimming videos. This method achieved a mean L2 marker error of 51.3 mm and enabled the successful real-world deployment of reinforcement learning policies on a physical fish, outperforming traditional black-box optimization techniques.
Key insights
- VLMs can effectively reason about complex physical discrepancies from video comparisons, translating visual observations into actionable parameter adjustments for high-dimensional robotic simulators.
- Combining iterative VLM feedback with a backtracking line search significantly enhances the efficiency and robustness of system identification by refining VLM proposals and increasing the rate of successful updates.
- Calibrating a simulator with VLM-guided system identification directly improves the performance and real-world transferability of downstream reinforcement learning policies for complex robotic platforms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.