Reproduce “Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Lee, Irfan Essa, Dhruv Batra
Semantic MapNet (SMNet) allows embodied AI agents to build allocentric (top-down) semantic maps from egocentric RGB-D observations. Developed by researchers at Georgia Institute of Technology, FAIR, and others, the model's 'Encode, Project, Segment' approach generated cleaner and more accurate maps, improving mean Intersection-over-Union by 4.01-16.81% over prior methods on Matterport3D.
Key insights
- Aggregating deep visual features in a spatial memory, rather than raw pixels or semantic labels, allows for more robust and accurate information integration over time and viewpoint changes.
- A Gated Recurrent Unit (GRU) can effectively learn to consolidate and smooth projected egocentric features into a coherent allocentric representation.
- The combination of precise projective camera geometry with a learnable feature aggregation and decoding mechanism addresses the limitations of previous semantic mapping paradigms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.