Reproduce “Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling” on OpenResearch

About this paper

William Xie, Max Conway, Yutong Zhang, Nikolaus Correll

A framework enables general-purpose Vision Language Models to directly generate wrenches for robot control, moving beyond traditional position-based manipulation to achieve a 51% zero-shot success rate on contact-rich tasks. The approach also reveals critical safety vulnerabilities, demonstrating how these capabilities can bypass VLM safeguards to generate harmful physical actions with up to an 87.9% harmful response rate for some models.

Key insights

• Vision Language Models (VLMs) possess latent physical understanding from their internet-scale training data, which can be successfully grounded into actionable robot commands for force-based control.
• Explicit visual coordinate frame labeling is crucial for grounding a VLM's abstract spatial reasoning into specific robot coordinate systems, with world-frame and aligned wrist-frame configurations proving most effective.
• The same physical reasoning capabilities that enable sophisticated robotic manipulation can be exploited to bypass current VLM safety safeguards, leading to the generation of harmful physical actions.

Summary by alphaXiv.