Reproduce “Unified Multimodal Visual Tracking with Dual Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingyi Hong, Jinglun Li, Xinyu Zhou, Kaixun Jiang, Pinxue Guo, Zhaoyu Chen, Runze Li, Xingdong Sheng and 1 more
A unified multimodal visual tracking framework, OneTrackerV2, handles diverse RGB and RGB+X inputs through a single, end-to-end trained model with shared parameters and a Dual Mixture-of-Experts (DMoE) architecture. It consistently improved performance across 12 tracking benchmarks and demonstrated enhanced robustness to missing modalities.
Key insights
- A learnable meta embedding can effectively mediate and unify heterogeneous multimodal features, making the system robust to missing data without direct concatenation.
- Decoupling expert learning into temporal (spatio-temporal matching) and multimodal (multimodal knowledge fusion) components within a DMoE architecture mitigates feature conflicts.
- Regularization techniques, specifically dissimilarity loss for experts and router clustering for multimodal experts, guide the DMoE to learn distinct and modality-coherent representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.