Reproduce “Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks” on OpenResearch

About this paper

Mirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada and 1 more

Mars-Bench introduces the first comprehensive benchmark for evaluating machine learning models on Mars science tasks, standardizing 20 datasets from orbital and surface imagery for classification, segmentation, and object detection. Baseline evaluations demonstrate that ImageNet-pretrained models often outperform Earth Observation-pretrained models on Mars tasks, highlighting a domain gap and the need for Mars-specific foundation models.

Key insights

• A significant domain gap exists between Earth and Mars imagery, as ImageNet-pretrained models often outperform Earth Observation models on Mars tasks, underscoring the need for Mars-specific foundation models.
• General-purpose Vision-Language Models lack the specialized geological knowledge required for accurate interpretation of complex Martian scientific concepts, performing inconsistently on expert-level tasks.
• Simpler architectures like U-Net can outperform more complex transformer-based models on Mars segmentation tasks when pre-trained on natural images, while object detection performance remains generally weak.

Summary by alphaXiv.