Reproduce “CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao An, Jiaxing Sun, Zihan Gui, Wei He
CHOICE, a new benchmark, objectively evaluates Large Vision-Language Models (VLMs) in remote sensing using 10,507 problems across 23 hierarchical tasks and exclusively self-collected images. Evaluations revealed that while specialized Remote Sensing VLMs (RSVLMs) perform well on domain-specific tasks, larger general-domain VLMs often achieve competitive or superior performance, with all models showing limitations in fine-grained perception and advanced reasoning.
Key insights
- RSVLMs excel in specialized remote sensing knowledge but do not consistently outperform larger general-domain VLMs, suggesting a need for integrating broader general knowledge.
- Fine-grained perception (e.g., object counting, visual grounding) and advanced reasoning (e.g., temporal properties, environmental assessment) present significant challenges across most evaluated VLMs.
- Top-tier open-source VLMs can achieve performance competitive with or superior to proprietary models in certain remote sensing tasks, offering viable alternatives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.