Reproduce “P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture” on OpenResearch

About this paper

Felix Tristram, Stefano Gasperini, Benjamin Killeen, Marcel Walch, Christian Benz, Nassir Navab, Ghazal Ghazaei

Researchers developed P-JEPA, a self-supervised framework that learns long-duration video representations for complex procedural tasks by predicting masked latent features. This approach achieves state-of-the-art fine-grained action classification on EgoExo4D with 56.85% top-1 accuracy and improves temporal action segmentation on Assembly101 and EgoProceL, while operating with significantly fewer parameters than LLM-based methods.

Key insights

• Self-supervised latent prediction on long video segments can effectively learn procedure-aware representations without explicit labels, narrations, or language models.
• 'Clip-causal attention,' which permits within-segment and past-segment attention but blocks future segments, is crucial for modeling procedural structure while enabling online inference.
• P-JEPA reorganizes the feature space, making visually similar actions contextually distinct and leading to feature trajectories that better align with temporal procedural progress.

Summary by alphaXiv.