Reproduce “Learning Object Placements For Relational Instructions by Hallucinating Scene Representations” on OpenResearch

About this paper

Oier Mees, Alp Emek, Johan Vertens, Wolfram Burgard

Researchers at the University of Freiburg developed a method enabling robots to interpret natural language spatial relations and place objects by estimating pixelwise placement probabilities from a single RGB image, using an auxiliary network and feature-level scene hallucination. This approach achieved mean Intersection over Union (IoU) scores of 0.63, 0.6, and 0.44 for predicted placement distributions and demonstrated an average human-robot object placement success rate of 84% in real-world experiments.

Key insights

• Learning pixelwise placement distributions can be achieved from weaker, more readily available relational bounding box annotations by leveraging an auxiliary task and feature-level hallucination, bypassing the need for explicit pixelwise ground truth.
• Implanting high-level features into an auxiliary network's feature maps, rather than manipulating pixels directly, effectively circumvents synthetic visual artifacts, leading to models that focus on meaningful spatial context.
• A robust relational classification network can serve as an effective supervisory signal for a placement prediction network, allowing complex spatial reasoning to be learned without direct pixel-wise ground-truth supervision.

Summary by alphaXiv.