Reproduce “Tonguescape: Exploring Language Models Understanding of Vowel Articulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haruki Sakajo, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
Researchers at the Nara Institute of Science and Technology introduced "Tonguescape," a novel dataset derived from real-time MRI recordings, to evaluate multi-modal language models' ability to understand vowel articulation from visual tongue positions. The study found that while models perform near chance in zero-shot visual tasks, their accuracy significantly improves in few-shot settings, indicating an ability to leverage relative tongue position cues.
Key insights
- Multi-modal LMs exhibit limited "embodied understanding" of vowel articulation from visual input alone in zero-shot scenarios, performing near random chance.
- Performance significantly improves in few-shot settings, demonstrating LMs' ability to infer vowels by considering "relative" tongue positions when reference examples are provided.
- The impact of explicit visual guides on images varies across models, aiding some models like GPT-4o and Qwen2-VL-72B-Instruct while acting as noise for others.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.