Reproduce “OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments” on OpenResearch

About this paper

Hymalai Bello, Lala Ray, Joanna Sorysz, Sungho Suh, Paul Lukowicz

The OpenMarcie dataset provides a comprehensive multimodal collection of human action data recorded in industrial settings, integrating synchronized wearable sensors, egocentric and exocentric multi-view video, and audio during complex assembly tasks. It facilitates advanced human activity recognition, open vocabulary captioning, and cross-modal alignment, demonstrating that multimodal fusion can achieve Macro F1 scores up to 0.915 for HAR and supports research into privacy-preserving AI.

Key insights

• Multimodal fusion, particularly combining vision and inertial sensors, consistently yields superior performance across human activity recognition, open vocabulary captioning, and cross-modal alignment tasks in industrial environments.
• Although privacy-preserving audio anonymization reduces general discriminative power, the acoustic modality remains highly effective for detecting specific, transient tool-contact events.
• Large Language Models (LLMs) can be reliably utilized for generating and validating structured activity labels from natural language descriptions, demonstrating their utility in complex dataset annotation pipelines.

Summary by alphaXiv.