Reproduce “STRMs: Spatial Temporal Reasoning Models for Vision-Based Localization Rivaling GPS Precision” on OpenResearch

About this paper

Hin Wai Lui, Jeffrey L. Krichmar

The paper introduces SpatioTemporal Reasoning Models (STRMs) that determine geographical coordinates using only visual input from a camera. These generative models transform first-person views into global map perspectives, achieving localization accuracy comparable to commercial GPS in certain environments, such as a university campus where the VAE-Transformer achieved a median deviation of 2.29 meters, outperforming phone GPS at 3.22 meters.

Key insights

• A generative approach, inspired by biological navigation, can directly transform first-person views into global map representations and precise geographical coordinates, eliminating the need for large satellite image databases.
• Transformer-based architectures (VAE-Transformer) are particularly effective for temporal reasoning in vision-based localization, outperforming RNNs and maintaining consistent accuracy over time.
• Reconstruction of the global map perspective (GMP) from first-person views serves as a powerful regularization mechanism, especially in complex urban environments, improving overall localization performance.

Summary by alphaXiv.