Reproduce “Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data” on OpenResearch

About this paper

Advait Pavuluri, Shamik Karkhanis, Uzma Mushtaque

Researchers at Rensselaer Polytechnic Institute evaluated a Joint-Embedding Predictive Architecture (JEPA) for autonomous driving clip triage, revealing that its high performance on cross-dataset benchmarks (AP 0.89) stems from detecting domain shifts rather than true event novelty. On a fair within-domain benchmark, the JEPA method yielded an AP of 0.29, despite a supervised probe showing the underlying V-JEPA 2 embeddings contained the review-worthiness signal (AP 0.50).

Key insights

• Cross-dataset evaluation protocols can lead to inflated performance metrics for novelty detection because the 'novelty score' often acts as a domain classifier.
• General-purpose self-supervised objectives, like the JEPA masking objective, may not effectively extract specific, rare-event 'triage signals' even when those signals are present in the underlying rich representations.
• Robust evaluation requires decorrelating review value from data provenance and utilizing supervised probes on frozen features to distinguish between absent signals and ineffective extraction by self-supervised methods.

Summary by alphaXiv.