Reproduce “SignMatch: Matching Dictionary Signs to Continuous Sign Language Video” on OpenResearch

About this paper

Ryan Wong, Youngjoon Jang, Liliane Momeni, Gül Varol, Andrew Zisserman

The SignMatch framework from the University of Oxford establishes a two-stage approach to visually match dictionary signs to segments in continuous sign language videos. It first learns a structured sign embedding space from continuous video data, then maps isolated dictionary signs into this space, achieving state-of-the-art performance on dictionary retrieval and sign spotting benchmarks, including cross-lingual transfer to unseen sign languages.

Key insights

• Decoupling representation learning from dictionary alignment yields a more robust and generalizable framework, especially when direct linguistic supervision is limited.
• Learning a structured embedding space with soft targets and prototypes effectively captures visual similarities and organizes sign classes, leading to better discriminative features.
• A purely visual matching approach can enable effective cross-lingual transfer, demonstrating that the learned representations capture universal visual properties of signs independent of specific languages.

Summary by alphaXiv.