Reproduce “Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation” on OpenResearch

About this paper

Yu Guo, Chang Yu, Siyu Ma, Yunuo Chen, Yin Yang, Ying Nian Wu, Chenfanfu Jiang

Researchers from UCLA, UCSD, and the University of Utah developed Sparse2Act, a pretraining framework that teaches sparse 3D encoders to learn action-aligned geometric features for robot manipulation. This framework effectively uses 3D end-effector actions as direct geometric supervision, leading to an 86.9% success rate on the LIBERO-10 benchmark and robust cross-domain and sim-to-real transfer with limited real-world data.

Key insights

• Robot actions can serve as a direct, control-grounded form of geometric supervision, effectively shaping sparse 3D encoders to learn reusable representations that are less tethered to specific downstream tasks or policy architectures.
• The combination of masking input tokens and an action-alignment objective is crucial for performance, as it forces the encoder to infer task-space actions from partial, visible geometric context, thereby organizing sparse geometric structures around 3D end-effector motion.
• The performance gains observed with Sparse2Act are largely attributable to the quality of the pretrained encoder's representation, making the 3D latent space more directly usable for control and reducing the reliance on high-capacity downstream decoders.

Summary by alphaXiv.