Reproduce “Tracking with Human-Intent Reasoning” on OpenResearch

About this paper

Jiawen Zhu, Zhi-Qi Cheng, Jun-Yan He, Chenyang Li, Bin Luo, Huchuan Lu, Yifeng Geng, Xuansong Xie

Researchers from Dalian University of Technology, Carnegie Mellon University, and Alibaba Group developed TrackGPT, an object tracker that interprets implicit human instructions by integrating large vision-language models. The framework defines a new "Instruction Tracking" task and achieves competitive performance on existing referring video object segmentation benchmarks, alongside state-of-the-art results on the newly introduced InsTrack benchmark.

Key insights

• Large Vision-Language Models can be effectively adapted with minimal fine-tuning (LoRA) and specialized tokens (<TK>, <PO>) to imbue traditional vision tasks with complex reasoning capabilities for human intent.
• Incorporating a self-correction or "rethinking" mechanism that re-engages the reasoning component based on tracking quality significantly enhances robustness and adaptability in dynamic video environments.
• A strong, pre-trained visual encoder (e.g., from SAM) is crucial for providing robust visual features, complementing the high-level reasoning of the LVLM brain for precise segmentation.

Summary by alphaXiv.