Reproduce “EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models” on OpenResearch

About this paper

Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, Richard Newcombe

Meta Reality Labs Research established EFM3D, a benchmark for 3D object detection and surface regression using egocentric data from wearable devices, contributing new datasets with 3D object bounding box annotations and ground-truth meshes. Their Egocentric Voxel Lifting (EVL) baseline model achieved 0.75 mAP on synthetic ASE and 0.22 mAP on real-world AEO for 3D object detection, and superior surface reconstruction quality, outperforming existing methods.

Key insights

• Temporal aggregation of per-snippet predictions via a simple object tracker significantly improves 3D object detection mean Average Precision (mAP) for egocentric video sequences.
• Integrating semi-dense point clouds as explicit geometric priors (point and freespace masks) within a 3D voxel representation substantially enhances model generalization and robustness for both 3D object detection and surface regression tasks.
• Leveraging features from a powerful pre-trained 2D foundation model (DinoV2.5) as the initial encoder, combined with a 3D U-Net, provides a robust basis for processing diverse egocentric modalities.

Summary by alphaXiv.