Reproduce “iSafetyBench: A video-language benchmark for safety in industrial environment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Raiyaan Abdullah, Yogesh Singh Rawat, Shruti Vyas

The iSafetyBench benchmark evaluates video-language models on routine and hazardous actions in short clips, including scenes with multiple actions. It contains 1,100 YouTube-sourced clips, 98 routine labels, and 67 hazard labels, assessed with single-choice and multi-choice questions. In zero-shot testing, Qwen2.5-VL-7B-Instruct scored 46.9% on normal and 33.6% on hazardous clips; hazards scored lower, without establishing why or deployment readiness.

Key insights

  • State-of-the-art Vision-Language Models demonstrate limited overall understanding in industrial safety environments, typically achieving F1 scores between 35% and 50% under zero-shot conditions.
  • Models perform consistently lower on hazardous actions compared to routine normal actions, with an average F1 score drop of 5%, indicating a significant challenge in recognizing rare, safety-critical events.
  • Performance varies by category, with higher scores on structured, object-centric normal actions and visually distinct hazards, but lower scores on subtle interactions and context-dependent hazards.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.