Reproduce “mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and 1 more
NVIDIA researchers developed `mindmap`, a 3D diffusion policy that integrates spatial memory by continuously building a metric-semantic 3D reconstruction of the environment. The system achieved a 76% average success rate on novel spatial memory tasks, demonstrating a 56% absolute improvement over baseline 3D diffusion policies lacking this memory capability.
Key insights
- Integrating a continuously updated metric-semantic 3D reconstruction provides persistent spatial memory, enabling policies to reason about objects not currently in the camera's field of view.
- Utilizing semantically rich features from a frozen Vision Foundation Model (AM-RADIO) is crucial for robust scene understanding and generalizability across different viewpoints and lighting conditions.
- Processing instantaneous observations and reconstructed spatial memory through distinct encoders allows the policy to effectively attend to both current, high-resolution details and accumulated environmental context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.