Reproduce “LipCoder: Voice-Enabled Coding Toolkit” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hayoon Kim, Sungho Lee, Juhwi Kim, Bongwon Suh, Kyogu Lee

LipCoder integrates speech commands, auditory code navigation, structural summaries, diagnostics, and LLM-assisted generation into a modular Visual Studio Code extension designed for blind and low-vision programmers. In a within-subjects study of five visually impaired developers, it raised mean SUS from 41.2 to 68.8, reduced NASA-TLX workload from 46.9 to 29.8, and increased overall task accuracy from 84.0% to 98.7%, although the corrected statistical tests did not establish significant differences.

Key insights

  • Auditory programming interfaces are most useful when they summarize structure and state—such as symbols, errors, and code changes—instead of reproducing every visual detail sequentially.
  • Participants broadly adopted high-level voice interaction for generation, explanation, navigation, and execution, while preferring text for precise edits, long prompts, and exact identifiers.
  • Audio preferences vary: delimiter and indentation cues were generally helpful, whereas multiple timbres for syntax elements were distracting for most participants, supporting configurable auditory presentation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.