Reproduce “Using Sign Language Production as Data Augmentation to enhance Sign Language Translation” on OpenResearch

About this paper

Harry Walsh, Maksym Ivashechkin, Richard Bowden

Researchers from the University of Surrey addressed data scarcity in Sign Language Translation (SLT) by generating synthetic sign language data using Sign Language Production (SLP) techniques. This approach, employing synthetic skeleton sequences via Sign Stitching and photo-realistic videos via SignSplat, improved Pose-to-Text translation BLEU-1 scores by up to 19% and Video-to-Text BLEU-4 scores by 0.44 points, making SLT models more robust.

Key insights

• Augmenting SLT datasets with synthetically generated skeleton sequences and photo-realistic videos effectively addresses data scarcity and significantly enhances translation performance.
• The quality of synthetic data is paramount; high-fidelity generation (e.g., SignSplat) leads to performance gains, while artifacts in lower-quality synthetic data (e.g., SignGAN) can be detrimental.
• Pre-training SLT models on augmented data, followed by fine-tuning on real data, is generally a more effective training strategy than joint training, especially for skeleton-based augmentation.

Summary by alphaXiv.