Reproduce “MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiqun Duan, Xianda Guo, Zheng Zhu, Zhen Wang, Yu-Kai Wang, Chin-Teng Lin
MaskFuser introduces a hybrid multi-modal sensor fusion network for end-to-end autonomous driving that creates a unified semantic feature space through joint tokenization and a cross-modality masked auto-encoder strategy. This approach improves driving performance on the CARLA LongSet6 benchmark by 4.5% and significantly enhances robustness to partial sensor failures.
Key insights
- Creating a unified semantic feature space through joint tokenization allows for deeper, more integrated understanding between different sensor modalities.
- A hybrid fusion approach, combining early geometric alignment (MBT attention) with late semantic integration (joint tokenization), effectively leverages the strengths of both.
- Applying a masked auto-encoder strategy for cross-modality reconstruction forces the model to learn robust and complementary feature representations, significantly improving resilience to partial sensor failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.