Reproduce “GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fang Liu, Jinpeng Chen, Ke Xu, Yuhao Liu, Huankang Guan, Xudong Lu, Bo Yang, Gerhard Hancke and 2 more
Researchers at City University of Hong Kong and Huawei Research developed GuideMe, a multi-domain streaming benchmark designed to evaluate Multimodal Large Language Models (MLLMs) as real-time procedural coaches. The benchmark reveals that current MLLMs struggle significantly with timely error detection and providing effective corrective guidance in interactive, closed-loop scenarios, exhibiting a noticeable performance gap compared to their ability to deliver routine instructions.
Key insights
- Current MLLMs exhibit a striking asymmetry in performance, reliably delivering basic instructions but significantly degrading in their ability to detect errors and provide corrective guidance during live procedural coaching.
- No zero-shot MLLM baseline consistently provided reliable streaming coaching, often failing to adapt from routine instructions to necessary error correction or showing poor trade-offs between intervention frequency and quality.
- General MLLM scaling and existing streaming-specific pretraining methodologies do not adequately instill the proactive intervention and error detection capabilities required for effective real-time procedural assistance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.