Reproduce “AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jirong Zha, Yuxuan Fan, Tianyu Zhang, Geng Chen, Yingfeng Chen, Chen Gao, Xinlei Chen
AirCopBench is introduced as the first comprehensive benchmark to evaluate Multimodal Large Language Models (MLLMs) in multi-drone collaborative embodied perception under challenging real-world conditions. The benchmark, featuring a dataset of 2.9k multi-view images and 14.6k VQA pairs across 14 tasks, reveals a significant performance gap between current MLLMs and human capabilities, while demonstrating that fine-tuning on the dataset substantially improves model performance and sim-to-real transferability for UAV applications.
Key insights
- Current MLLMs exhibit a substantial performance gap (24.38% behind human baseline) in multi-drone collaborative embodied perception, indicating significant room for improvement.
- The ability to perform "Causal Assessment" (inferring reasons for perception degradation) shows a high correlation with overall performance, suggesting it is a fundamental cognitive capability for embodied perception.
- Supervised fine-tuning on AirCopBench significantly boosts MLLM performance (e.g., Qwen-2.5-VL-7B improved by 26.97%), and sim-to-real transfer is viable, achieving a 19.64% increase in real-world accuracy on average.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.