Reproduce “Towards Online Continuous Sign Language Recognition and Translation” on OpenResearch

About this paper

Ronglai Zuo, Fangyun Wei, Brian Mak

Researchers at HKUST and Microsoft Research Asia developed an online framework for Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT) that processes sign videos in real-time. This system achieves new state-of-the-art performance on multiple benchmarks while reducing algorithmic latency by 92% and memory cost by 66% compared to offline methods.

Key insights

• The pseudo ground truth generation via CTC forced alignment and subsequent sign augmentation effectively addresses the lack of pre-segmented isolated sign data for continuous sign language benchmarks, aligning training with real-time inference scenarios.
• The combination of gloss-level sampling and multi-component loss functions (instance, gloss, and saliency losses) enables the ISLR model to learn robust and separable representations for individual signs, effectively handling co-articulations and improving classification accuracy.
• The online post-processing algorithm, featuring voting-based deduplication and background elimination, is crucial for transforming raw, window-based predictions into coherent and accurate sequences of recognized signs, ensuring smooth and reliable real-time output.
• The learned features from the online ISLR model are transferable and can enhance the performance of existing state-of-the-art offline CSLR models through a lightweight adapter network, demonstrating the versatility and quality of the online system's representations.

Summary by alphaXiv.