Reproduce “When Drawing Is Not Enough: Exploring Spontaneous Speech with Sketch for Intent Alignment in Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiyan Shi, Dorien Herremans, Kenny Tsu Wei Choo
TalkSketchD captures sketching and concurrent speech during toaster ideation, pairing continuous sketches with time-aligned transcripts and intent summaries from 11 participants across 58 instances. In a controlled image-generation comparison, with each sketch held constant, adding its unedited transcript produced higher model-judged intent-alignment scores: median scores were 7 versus 2 on each of four measures. This evidence comes from one toaster task and one generation-and-judge setup, not human ratings or a result established for other products or models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.