Reproduce “The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yizhak Ben-Shabat, Xin Yu, Fatemeh Sadat Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, Stephen Gould

Researchers from the Australian National University and City University of Hong Kong introduced the IKEA ASM dataset, a multi-modal, multi-view collection of 35 hours of furniture assembly videos with detailed annotations for fine-grained actions, human pose, and object instance segmentation. The dataset provides benchmarks for various computer vision tasks, revealing the challenges posed by complex, structured human activities in real-world settings.

Key insights

  • Combining multi-modal (RGB, depth, human pose) and multi-view data consistently improves action recognition performance in complex assembly tasks, outperforming single-modality or single-view approaches.
  • The dataset's real-world characteristics, including visually similar objects, uncommon human poses, and dynamic backgrounds, significantly challenge current state-of-the-art computer vision models, resulting in notably lower performance compared to generic datasets.
  • Detailed, fine-grained temporal and spatial annotations, particularly at the object instance and multi-joint human pose levels, are crucial for advancing holistic understanding of structured human activities and complex human-object interactions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.