Reproduce “OmniSAT: Compact Action Token, Faster Auto Regression” on OpenResearch

About this paper

Huaihai Lyu, Chaofan Chen, Senwei Xie, Pengwei Wang, Xiansheng Chen, Shanghang Zhang, Changsheng Xu

OMNISAT introduces a two-stage action tokenization pipeline for Vision-Language-Action models, enabling highly compressed yet high-fidelity discrete action representations. This approach accelerates auto-regressive training and yields state-of-the-art performance on various real-world and simulated robotic manipulation tasks, facilitating effective cross-embodiment learning.

Key insights

• Transforming variable-length continuous trajectories into fixed-length B-spline control points provides a numerically consistent and temporally aligned representation, crucial for cross-embodiment consistency.
• Physically partitioning action dimensions (e.g., position, rotation, gripper) and applying grouped residual vector quantization allows for more effective compression and higher reconstruction accuracy by exploiting inherent action structure.
• A unified action token space, derived from the tokenizer, facilitates a cross-embodiment learning strategy, enabling VLA policies to effectively learn from diverse robot and human demonstration data.

Summary by alphaXiv.