Reproduce “Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation” on OpenResearch

About this paper

Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, Vincent Sitzmann

Neural Descriptor Fields (NDFs) enable robots to learn complex manipulation tasks from as few as 5-10 demonstrations, achieving robust generalization to unseen object instances in arbitrary 6-degree-of-freedom poses. The method leverages neural implicit representations to create SE(3)-equivariant descriptors, significantly outperforming 2D-based methods by maintaining up to 0.78 success rates on randomly oriented objects in simulation while baselines dropped to near-zero.

Key insights

• Leveraging internal activations of a self-supervised pre-trained occupancy network allows creating dense, continuous 3D spatial descriptors that capture hierarchical geometric features.
• Full 6-DoF poses can be encoded as SE(3)-equivariant descriptors by applying the pose to a set of predefined query points and concatenating their learned point descriptors.
• Achieving SE(3)-equivariance through mean-centering inputs and using SO(3)-equivariant network architectures guarantees generalization to arbitrary rigid body transformations of objects.

Summary by alphaXiv.