Reproduce “Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ozge Mercanoglu Sincan, Jian He Low, Sobhan Asasi, Richard Bowden
Mercanoglu Sincan et al. systematically re-evaluated five state-of-the-art gloss-free Sign Language Translation models under unified conditions, revealing that many previously reported performance gains diminish without specific implementation choices, while identifying cross-lingual contrastive learning as a robust contributor to improved translation quality.
Key insights
- A substantial portion of reported performance gains in gloss-free SLT is often tied to specific implementation details and evaluation setups rather than solely to core methodological innovations.
- Certain components, such as frame-level contrastive learning (SignCL) and especially cross-lingual contrastive learning (C²RL's CiCO loss), consistently improve translation quality under unified conditions.
- Variations in evaluation metric calculations, including dataset-specific conventions and differences between libraries, significantly influence reported scores and hinder fair comparisons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.