Reproduce “ManipulaTHOR: A Framework for Visual Object Manipulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi
Researchers at the Allen Institute for AI introduced ManipulaTHOR, an extension to the AI2-THOR simulator, to enable embodied agents to perform low-level object manipulation in visually complex, physics-enabled environments. They developed the ArmPointNav task as a benchmark, achieving a 39.4% success rate in picking up and relocating objects without disturbance in previously seen scenes with familiar objects, while also demonstrating moderate generalization to novel objects.
Key insights
- An end-to-end deep reinforcement learning approach is more effective for complex combined navigation and manipulation tasks like ArmPointNav than disjoint multi-stage policies, which struggle with overall task completion despite specialized sub-task performance.
- Depth information is more critical for spatial reasoning, collision avoidance, and overall task success in visual object manipulation compared to RGB information.
- Generalization to novel object categories presents a greater challenge than generalization to novel scenes, suggesting that robust understanding of object geometry and interaction properties is a primary bottleneck for manipulation tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.