Reproduce “Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy

The authors introduce a language-feedback policy (LFP) to steer a frozen vision-language-action model by selecting task-relevant instructions through closed-loop rollouts and learning when to fall back to the original instruction. In a cube-sorting episode, it recovers after a placed cube is moved back. In simulation, conformalized refusal raises success to 75.96% and lowers harmful-case FPR to 9.31%, conditional on exchangeability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.