Reproduce “FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Garrett Tanzer
FLEURS-ASL adds carefully prepared American Sign Language video translations to the FLORES/FLEURS multilingual benchmark family. Five Certified Deaf Interpreters produced 495 discourses and 1,749 sentences, totaling 7.49 hours. A unified landmark-based model scored 3.7 BLEU and 37.2 BLEURT zero-shot on ASL-to-English translation. Three tested multimodal models performed near zero, within this setup.
Key insights
- High-quality sign language benchmarks require rigorous data collection methods, including the use of Certified Deaf Interpreters (CDIs) and ample preparation time for culturally and linguistically accurate translations.
- Integrating long-range contextual information, both visual and textual, significantly aids in improving sign language translation model performance beyond isolated sentence-level approaches.
- Current state-of-the-art large multimodal models demonstrate virtually no understanding of American Sign Language, highlighting a critical gap in their generalized capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.