Reproduce “CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations” on OpenResearch

About this paper

Alexandra Carlson, Manikandasriram Srinivasan Ramanagopal, Nathan Tseng, Matthew Johnson-Roberson, Ram Vasudevan, Katherine A. Skinner

CLONeR is a camera-LiDAR fusion framework for Neural Radiance Fields that models large, unbounded outdoor driving scenes from sparse sensor inputs. It decouples geometry and color learning, supervising geometry with LiDAR and color with camera images, and employs a differentiable Occupancy Grid Map for efficient sampling, achieving a μSILog of 0.101 for depth prediction on KITTI, markedly outperforming baselines.

Key insights

• Decoupling geometry and color learning within a NeRF framework, with appropriate sensor supervision, prevents image artifacts from corrupting 3D scene structure in challenging outdoor environments.
• A learned, differentiable Occupancy Grid Map provides a more efficient and accurate way to guide volumetric rendering in large, sparse scenes by focusing sampling on occupied regions.
• Effective fusion of complementary sensor data (camera for texture, LiDAR for depth) enables robust 3D scene reconstruction and novel view synthesis even from very sparse input views.

Summary by alphaXiv.