Reproduce “What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chi-Hsi Kung, Frangil Ramirez, Juhyung Ha, Yi-Ting Chen, David Crandall, Yi-Hsuan Tsai
Researchers at Indiana University developed a hierarchical framework for procedure-aware video representation learning that leverages Large Language Models to generate state-change descriptions and counterfactuals as supervision. This approach enhances video encoders' ability to understand both actual and hypothetical scene evolutions, leading to improved performance in tasks such as temporal action segmentation and unsupervised error detection.
Key insights
- Explicitly modeling both 'what changed' (actual state transitions) and 'what could have changed' (hypothetical failure outcomes) through LLM-generated textual supervision significantly enhances a model's causal understanding of procedural videos.
- Incorporating diverse counterfactual types during pre-training (state-change, missing-step, misordered) provides models with exposure to potential deviations, which improves their robustness and ability to detect errors in procedural activities.
- A hierarchical learning approach, combining fine-grained state-change understanding with long-range procedural awareness from counterfactuals, yields generalizable video representations effective across a wide range of downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.