Reproduce “AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Enmin Zhong, Carlos R. del-Blanco, Daniel Berjón, Fernando Jaureguizar, Narciso García

AnimalMotionCLIP, from Universidad Politécnica de Madrid, enhances animal behavior recognition by integrating optical flow with video frames into a CLIP-based visual-language model. This framework achieves a mean average precision of 74.63% on the Animal Kingdom dataset, outperforming existing state-of-the-art methods, including those reliant on extensive pre-training on human-centric datasets.

Key insights

  • Explicitly combining optical flow with RGB frames provides complementary information that significantly improves animal behavior recognition accuracy, moving beyond reliance on static visual features.
  • A sparse temporal sampling strategy, which quasi-randomly selects frames from the entire video, proves more effective for capturing diverse behavioral contexts than dense or semi-dense local sampling.
  • Pre-trained visual-language models can be effectively adapted for specialized domains like animal behavior without the need for additional, computationally intensive pre-training on large, human-centric action datasets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.