Reproduce “Emergent 3D Instance Segmentation from Self-Supervised Point Transformers” on OpenResearch

About this paper

Ted Lentsch, Santiago Montiel-Marín, Holger Caesar, Julian F. P. Kooij

Researchers demonstrate that self-supervised 3D point transformers implicitly learn to distinguish object instances, with the emergent instance segmentation signal residing in the attention queries and keys. A training-free method, TokenGraph3D, leverages these internal representations to achieve competitive unsupervised 3D instance segmentation performance on outdoor LiDAR datasets.

Key insights

• The instance grouping signal is concentrated in the attention queries and keys of the self-supervised point transformer, rather than in the values or final output features, aligning with observations in 2D vision transformers.
• Attention keys maintain a robust instance signal across varying object sizes, while the final output feature exhibits a collapse in signal for larger objects due to a tendency to merge neighbors.
• The instance signal demonstrates a bimodal strength distribution across encoder depths, being strongest at both the shallowest and deepest stages of the transformer.
• The Rotary Position Encoding (RoPE) is identified as the primary computational mechanism driving the emergence of instance-discriminative capabilities within the keys.

Summary by alphaXiv.