Reproduce “Temporal Segment Transformer for Action Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhichao Liu, Leshan Wang, Desen Zhou, Jian Wang, Songyang Zhang, Yang Bai, Errui Ding, Rui Fan
The Temporal Segment Transformer (TST) improves action segmentation by jointly modeling temporal relationships between action segments and denoising their features within a "predict-refine" framework. This approach establishes new state-of-the-art performance across the 50Salads, GTEA, and Breakfast datasets.
Key insights
- Jointly performing segment-level feature denoising and temporal relation modeling is crucial for overcoming the limitations of noisy initial segment representations in action segmentation.
- Local segment-frame attention, by focusing on a restricted temporal window, proves more effective for denoising segment features compared to global attention, indicating the importance of localized contextual refinement.
- Both the aggregated visual features from frames (frame embedding) and the initial predicted action category (category embedding) contribute meaningfully to forming robust initial segment representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.