Reproduce “The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yumeng Shi, Quanyu Long, Yin Wu, Wenya Wang
This work introduces VT-Contrast, a method that directly supervises video-token representations in Video-Language Models to enhance their temporal understanding. It trains models to differentiate between original and temporally reordered video sequences, leading to consistent performance gains on benchmarks requiring event progression reasoning.
Key insights
- Directly shaping video-token representations with temporal counterfactuals is more effective for instilling intrinsic temporal understanding than relying solely on indirect text-level supervision.
- Targeting supervision to late-layer last-frame video tokens is optimal, as these features tend to aggregate comprehensive temporal cues across the video sequence.
- Grading the difficulty of temporal counterfactuals, specifically by focusing on 'harder' negatives using Kendall tau distance, improves the model's ability for fine-grained temporal discrimination.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.