Reproduce “TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching” on OpenResearch

About this paper

Chongjian Wang, Junjie Gao

TeaMatch introduces a training-only framework for 2D–3D matching that enhances the robustness and geometric consistency of cross-modal representations by enforcing "teachability." This approach, which involves using weak learners to recover degraded representations, improved registration recall by 9.4% on 7-Scenes compared to 2D3D-MATR and by 2.6% on RGB-D Scenes V2 over Diff2I2P, without adding inference costs.

Key insights

• Explicitly enforcing "teachability"—the ability of a representation to be effectively recovered by weak learners under degraded inputs—leads to more robust and structurally consistent cross-modal representations.
• Regularizing representations by testing their recoverability with constrained probes and diverse degradation patterns prevents the primary matcher from relying on fragile, context-dependent features, guiding it to learn inherently stable geometric structures.
• Aligning the teachability objective with downstream tasks through specific losses (inlierness, pose) ensures that the enhanced representation quality directly translates to more reliable correspondence selection and improved pose estimation accuracy.

Summary by alphaXiv.