Reproduce “ASFormer: Transformer for Action Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fangqiu Yi, Hongyu Wen, Tingting Jiang
Researchers at Peking University developed ASFormer, a Transformer-based model designed for action segmentation that adapts the architecture to overcome limitations with data scarcity and long video sequences. ASFormer achieved state-of-the-art results on public datasets including 50Salads, GTEA, and Breakfast by incorporating task-specific inductive biases and an iterative refinement mechanism.
Key insights
- Integrating local connectivity inductive bias through dilated temporal convolutions is crucial for Transformers to perform effectively on action segmentation datasets with limited training data.
- A hierarchical representation pattern, where attention window sizes double at each successive layer, efficiently handles long video sequences and guides self-attention to meaningful temporal contexts.
- An iterative refinement process, implemented with multiple decoders and a specific cross-attention design, significantly enhances the temporal consistency and accuracy of action boundaries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.