Reproduce “CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Jikai Wang, Qifan Zhang and 5 more

Researchers from UT Dallas and the University of Florida introduced CaptainCook4D, a multimodal egocentric 4D dataset designed to train and evaluate AI systems in recognizing errors during real-world cooking activities. Benchmarking on the dataset revealed that supervised error recognition achieved up to 70.41% accuracy with multimodal features, yet current AI models still face challenges in robustly identifying and localizing errors.

Key insights

  • Leveraging multimodal features (video, audio, depth, text) significantly improves supervised error recognition performance, with models combining all modalities outperforming single-modality approaches.
  • State-of-the-art Visual Language Models (VLMs) perform poorly on zero-shot error recognition tasks within long, egocentric procedural videos, indicating current limitations in complex semantic understanding and reasoning about errors.
  • Current multi-step localization methods exhibit a substantial drop in performance when applied to erroneous procedures compared to correct ones, highlighting their lack of robustness to deviations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.