Reproduce “You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences” on OpenResearch

About this paper

Ninad Daithankar, Alexi Gladstone, Yann LeCun, Heng Ji

Researchers from UIUC and NYU propose Temporal Difference in Vision (TDV), a self-supervised method for learning visual representations from video by predicting future frame embeddings from past frames and learned motion. This approach, which avoids strong inductive biases, demonstrates improved performance on dense spatial tasks like optical flow and stereo depth compared to existing techniques like DINO and iBOT.

Key insights

• The optimal strength of inductive biases in visual representation learning empirically decreases as the dataset size expands, favoring approaches with fewer hard-coded assumptions.
• The fundamental causal principle of predicting the immediate future from the past offers a weak yet universally valid inductive bias for learning visual representations from video.
• Factorizing video learning into frame content and motion in the latent space allows for explicit modeling of temporal dynamics, which is crucial for tasks requiring fine-grained spatial and temporal understanding.

Summary by alphaXiv.