Reproduce “Uncertainty-aware sign language video retrieval with probability distribution modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan Wu, Hongxiang Li, Yuanjiang Luo, Xuxin Cheng, Xianwei Zhuang, Meng Cao, Keren Fu
This paper presents UPRet, a new approach for sign language video retrieval that models both sign language videos and text queries as probability distributions to account for the inherent ambiguity in sign languages. By employing optimal transport theory for distribution alignment, the method achieves improved retrieval accuracy, surpassing existing methods on benchmark datasets such as How2Sign, PHOENIX-2014T, and CSL-Daily.
Key insights
- Modeling sign language videos and textual queries as probability distributions, rather than single points, can effectively capture and manage the inherent uncertainty and ambiguity in sign language communication.
- Optimal transport theory offers a robust method for measuring similarity between these probability distributions, providing a more accurate alignment than traditional point-based distance metrics.
- Combining domain-agnostic and domain-aware features within the sign encoder enhances the system's ability to understand both general video content and specific sign language nuances.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.