Reproduce “RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanqiu Yu, Zhifan Jin, Sijia Chen, Tongfei Chu, En Yu, Liman Liu, Wenbing Tao
Researchers from Huazhong University of Science and Technology and South-Central Minzu University introduce RT-RMOT, a new task for robust all-day multi-object tracking that fuses language instructions with RGB and thermal visual data. They present the RefRT dataset and the RTrack framework, which improved the HOTA metric by 6.84% on the new dataset compared to adapted methods.
Key insights
- A deep fusion of language, RGB, and thermal modalities is essential for achieving robust, all-day referring multi-object tracking, leveraging complementary strengths of each data type.
- Multimodal Large Language Models (MLLMs), specifically Qwen2.5-VL, can serve as effective foundational models for complex cross-modal tasks like RT-RMOT due to their strong multimodal perception and instruction understanding capabilities.
- Reinforcement learning fine-tuning, when stabilized by strategies like Clipped Advantage Scaling (CAS) and guided by structured/comprehensive reward functions, significantly enhances performance and stability in MLLM-based vision-language tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.