Reproduce “Masked Feature Prediction for Self-Supervised Visual Pre-Training” on OpenResearch

About this paper

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, Christoph Feichtenhofer

MaskFeat introduces a self-supervised pre-training method for visual models that masks portions of inputs and predicts target features, achieving 86.7% top-1 accuracy on Kinetics-400 for video recognition with MViT-L using only unlabeled video data. The approach demonstrates that predicting Histograms of Oriented Gradients (HOG) is particularly effective for this task, offering a simpler and more efficient alternative to previous methods.

Key insights

• Directly regressing continuous feature representations, such as Histograms of Oriented Gradients (HOG), is more effective and simpler for masked visual prediction than relying on discrete visual vocabularies.
• Classic hand-crafted features like HOG can serve as robust and efficient self-supervision targets for modern deep learning models, outperforming more complex deep features or pixel-level targets.
• Effective pre-training for large Transformer-based video models can be achieved solely on unlabeled video data, significantly reducing the dependence on costly supervised image datasets and leading to improved transferability across diverse video tasks.

Summary by alphaXiv.