Reproduce “Unsupervised Video Representation Learning by Bidirectional Feature Prediction” on OpenResearch

About this paper

Nadine Behrmann, Juergen Gall, Mehdi Noroozi

Researchers from Bosch Center for Artificial Intelligence and University of Bonn developed Bidirectional Feature Prediction, a self-supervised method for learning video representations that concurrently predicts unobserved past and future frames. This technique, utilizing temporal hard negatives, achieved improved performance in action recognition, reaching 66.4% top1 accuracy on UCF101 and 45.3% on HMDB51, using only RGB video input.

Key insights

• Jointly predicting past and future frames provides a richer and more comprehensive supervisory signal for learning temporal dynamics than relying solely on future prediction.
• The use of structured temporal hard negatives, derived from temporal order swapping, is critical for robust representation learning, enabling the model to learn nuanced temporal ordering beyond distinguishing random noise.
• Independent spatial augmentations foster the learning of global, spatially invariant features that are more beneficial for high-level tasks in deeper network layers, as shown in layer-wise evaluations.

Summary by alphaXiv.