Reproduce “AnyTrack: Unifying Visual Object Tracking with Any Modalities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hao Li, Yunzhi Zhuge, Wenning Hao, Pingping Zhang, Xiaoxiong Zhang, Dong Wang, Huchuan Lu

AnyTrack introduces a unified framework for visual object tracking that processes arbitrary combinations of diverse modalities, including non-visual cues like language and audio, within a single model. This framework achieves superior performance across multiple benchmarks, demonstrating robustness in scenarios with missing or degraded sensor data.

Key insights

  • Dynamic interaction via a Mixture-of-Experts (MoE) framework effectively bridges modality discrepancies and maintains spatio-temporal consistency across diverse inputs.
  • Integrating multi-modal global-local prompts (language, audio, motion) significantly enhances target discrimination and refines localization by providing rich contextual guidance.
  • A unified model can robustly handle any combination of diverse, and even incomplete, modalities, demonstrating adaptability beyond fixed-modality designs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.