Reproduce “Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo Suzuki

Researchers from the University of Colorado Boulder developed "Guided Reality," an AR system that uses large language models and vision models to automatically generate and embed visually-enriched, spatially-grounded task guidance in real-time. This system improves user task performance by reducing error rates (e.g., from 17.14% to 10.42% for printer cleaning) and provides a highly usable experience (SUS score of 84.38).

Key insights

  • A taxonomy of visually rich AR guidance, derived from user manual analysis, significantly enhances task comprehension and performance compared to basic highlights.
  • The combination of LLMs for semantic understanding and vision models for spatial grounding enables robust, on-demand generation and accurate embedding of complex visual instructions without manual pre-authoring.
  • Different types of visual guidance have varying effectiveness, with dynamic movement indications and interactive widgets being particularly favored by users for conveying complex actions or providing context.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.