Reproduce “OBSR: Open Benchmark for Spatial Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Julia Moska, Oleksii Furman, Kacper Kozaczko, Szymon Leszkiewicz, Jakub Polczyk, Piotr Gramacki, Piotr Szymański

The OBSR benchmark provides a standardized, multi-task, and modality-agnostic framework for evaluating geospatial embedders and GeoAI models across 7 diverse datasets and 5 downstream tasks. Initial evaluations highlight the impact of spatial resolution on model performance and the current limitations of OpenStreetMap-based embeddings for dynamic trajectory predictions.

Key insights

  • Geospatial model performance varies significantly with spatial resolution, revealing challenges in building models that generalize across different spatial scales.
  • Tasks characterized by skewed target distributions, such as crime activity prediction, pose learning difficulties and necessitate careful interpretation of performance metrics.
  • Existing OpenStreetMap (OSM) based embeddings combined with basic sequence models demonstrate limitations in effectively capturing dynamic spatial relationships for trajectory prediction tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.