Reproduce “SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning” on OpenResearch

About this paper

Jianhe Low, Alexandre Symeonidis-Herzig, Maksym Ivashechkin, Ozge Mercanoglu Sincan, Richard Bowden

This research introduces SignSparK, a Conditional Flow Matching framework that generates fluid, 3D multilingual sign language motion by conditioning on sparse keyframes and spoken text. The system, supported by an efficient keyframe segmentation model called FAST, achieves state-of-the-art performance in various Sign Language Production tasks and demonstrates 100x faster inference.

Key insights

• Sparse, linguistically defined keyframes effectively anchor motion generation, preventing "regression-to-the-mean" and enabling the learning of fluid coarticulation dynamics between signs.
• Focusing on unimodal hand-centric features (WiLoR parameters) for segmentation provides significant efficiency gains (45x faster hand detection) without compromising accuracy, enabling scalable keyframe extraction.
• A reconstruction-based Conditional Flow Matching formulation drastically enhances inference efficiency, achieving over 100x acceleration compared to prior CFM models, while maintaining robust performance in low-step sampling regimes.

Summary by alphaXiv.