Reproduce “Learning Object Permanence from Videos via Latent Imaginations” on OpenResearch

About this paper

Manuel Traub, Frederic Becker, Sebastian Otte, Martin V. Butz

Loci-Looped, a deep learning system developed by researchers at the University of Tübingen, autonomously acquires fundamental intuitive physical concepts like object permanence and directional inertia directly from video data. It accomplishes this by adaptively fusing external sensory observations with internal latent imaginations, resulting in a mean tracking error of 2.6 pixels and 0.84 Multiple Object Tracking Accuracy (MOTA) on the ADEPT dataset, significantly outperforming baselines in handling occlusions and sensory interruptions.

Key insights

• Self-supervised learning, without explicit labels, can enable deep learning systems to acquire complex intuitive physical concepts like object permanence and directional inertia from observational video data.
• An adaptive, object-specific fusion mechanism between internal latent imaginations and external sensory observations is crucial for robustly tracking and understanding objects through extended occlusions and sensory interruptions.
• Encouraging sparse reliance on external observations via L0 regularization fosters the development of a strong internal world model, improving generalization to prolonged occlusions and enabling anticipatory responses to physical events.

Summary by alphaXiv.