Reproduce “Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianyuan Guo, Peike Li, Trevor Cohn
The PGG-SLT framework introduces a method for generating and refining pseudo glosses using large language models and weakly supervised learning to enhance gloss-free sign language translation. It demonstrated improvements in BLEU4 scores by up to 5.0 on Phoenix14T and 6.1 on How2Sign compared to prior gloss-free methods, achieving performance competitive with gloss-based approaches.
Key insights
- Large Language Models can serve as effective 'surrogate experts' for generating specialized linguistic annotations, such as glosses, with minimal in-context examples.
- Structured intermediate representations, even if pseudo-generated, are crucial for bridging the modality gap between visual sign languages and spoken languages in a gloss-free context.
- Weakly supervised methods can effectively refine and temporally align automatically generated pseudo glosses with video content, improving their utility for sequence-alignment losses like CTC.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.