Reproduce “ReferTrack: Referring Then Tracking for Embodied Visual Tracking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang, Zibo Zhang, Weisi Situ, Yuchen Zhou, Yonggen Ling and 1 more

ReferTrack introduces a "referring then tracking" paradigm for embodied visual tracking, directly grounding natural language instructions to image-space bounding box detections within a VLA model. This approach achieves state-of-the-art single-view performance on the EVT-Bench benchmark, particularly improving success rates on distracted (+6.8%) and ambiguity tracking (+22.9%) tasks, and demonstrates robust sim-to-real transfer on physical robots.

Key insights

  • Explicit image-space target identification, via the Refer-CoT mechanism, is a primary driver for robustness against distractors and ambiguities, and can compensate for limitations of single-camera setups.
  • Target identification, rather than motion planning, is the main bottleneck in complex distracted tracking environments, as evidenced by significant performance gains when provided with ground-truth target bounding boxes.
  • Integrating temporal bounding box geometry into visual history via TVBI tokens effectively provides target-specific memory, which contributes to improved tracking stability and success over time.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.