Reproduce “Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining” on OpenResearch

About this paper

Yanchen Wang, Han Yu, Ari Blau, Yizi Zhang, International Brain Laboratory, Liam Paninski, Cole Hurwitz, Matt Whiteway

BEAST learns representations from unlabeled experimental videos through masked reconstruction and temporal contrastive learning, pairing each frame with a neighbor and treating other batch frames as negatives. It supports neural encoding, pose estimation, and action classification. On held-out CalMS21 animals, its patch-based TCN reaches macro-F1 0.81 ± 0.01 versus TREBA’s 0.72 ± 0.01; action training still uses labels, while validation across environments and species remains needed.

Key insights

• Domain-specific self-supervised pretraining on unlabeled animal behavioral videos yields more effective representations than general computer vision foundation models for neuroscientific tasks.
• Combining Masked Autoencoding for fine-grained spatial appearance with temporal contrastive learning for dynamics is critical for learning versatile features adaptable to diverse downstream applications.
• Raw video data, when processed by models like BEAST, provides richer and more effective behavioral features for predicting neural activity compared to traditional sparse keypoint tracking methods.

Summary by alphaXiv.