Reproduce “ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation” on OpenResearch

About this paper

Dayoung Gong, Suha Kwak, Minsu Cho

ActFusion, a unified diffusion model developed by POSTECH, simultaneously performs temporal action segmentation (TAS) and long-term action anticipation (LTA) using a single architecture and training process. It achieves state-of-the-art performance on multiple benchmarks for both tasks, demonstrating bidirectional benefits between segmentation and anticipation, and addresses previous LTA evaluation protocol issues.

Key insights

• Temporal action segmentation and long-term action anticipation are inherently complementary tasks, yielding bidirectional performance benefits when learned jointly within a unified model.
• Diffusion models, when equipped with anticipative and random masking strategies, can effectively process and predict both observed and unobserved portions of temporal action sequences.
• The use of learnable mask tokens and their early integration into the encoder significantly enhances the model's ability to anticipate actions and improves overall task performance.

Summary by alphaXiv.