Reproduce “End-to-End 3D Dense Captioning with Vote2Cap-DETR” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, Gang YU
Vote2Cap-DETR, developed by researchers from Fudan University, I2R A*STAR, Tencent PCG, and Sichuan University, introduces a one-stage, end-to-end transformer-based framework for 3D dense captioning, which directly predicts object bounding boxes and their natural language descriptions. The model achieves state-of-the-art performance, with up to an 11.13% absolute improvement in CIDEr score on the ScanRefer dataset, by integrating vote queries for improved localization and a dual-clued captioner for context-rich descriptions.
Key insights
- A one-stage, fully attention-driven transformer architecture can effectively integrate 3D object detection and caption generation, eliminating the need for complex, hand-crafted, multi-stage pipelines.
- Incorporating vote queries with both spatial bias and content-aware features significantly improves object localization and accelerates training convergence in transformer-based 3D detectors.
- Caption generation benefits significantly from leveraging local 3D contextual information around an object, enabling more descriptive and spatially aware sentences compared to relying solely on object-centric features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.