Reproduce “Decomposition Betters Tracking Everything Everywhere” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Li, Dong Liu
DecoMotion introduces a test-time optimization method that explicitly decomposes video content into static scenes and dynamic objects for improved long-range pixel tracking. This approach, which uses specialized neural representations and transformations for each component, achieves a 6.9% increase in average tracking position accuracy (δx_avg) on the TAP-Vid-DAVIS-Strided benchmark compared to OmniMotion.
Key insights
- Explicitly decomposing video content into static and dynamic components, rather than using a unified model, significantly enhances the accuracy and consistency of long-range point tracking.
- Leveraging discriminative 3D features (e.g., from DINO) through a feature rendering loss is crucial for rectifying non-rigid transformations and overcoming challenges like appearance changes and photometric distractors in dynamic object tracking.
- Including a small number of non-linear layers in the static transformation improves robustness by compensating for imperfect static/dynamic classification, while more layers in dynamic transformations are essential for complex non-rigid motion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.