Reproduce “Controllable Augmentations for Video Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Qian, Weiyao Lin, John See, Dian Li
Researchers from Shanghai Jiao Tong University and The Chinese University of Hong Kong developed a self-supervised framework for video representation learning that mitigates static background bias and enhances global temporal understanding. The method, which uses controllable augmentations for region contrast, low-level shortcut elimination, and temporal dependency, achieved 71.2% linear probe accuracy on UCF-101 and 25.4% Top-1 accuracy on the motion-centric Diving-48 dataset, surpassing existing RGB-only baselines.
Key insights
- Controllable spatio-temporal and low-level augmentations are effective in addressing specific biases in video representation learning, allowing for fine-grained control over positive pairs.
- Minimizing mutual information between features from similar low-level augmented samples helps decouple low-level statistics, forcing the model to learn more semantically meaningful features.
- Jointly leveraging local clip and global video information, coupled with temporal dependency modeling, is crucial for capturing accurate motion patterns and comprehensive long-term temporal structures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.