Reproduce “The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Litao Yan, Andrew Head, Ken Milne, Vu Le, Sumit Gulwani, Chris Parnin, Emerson Murphy-Hill
Researchers from the University of Pennsylvania and Microsoft developed InvisibleMentor, a system that infers user actions from screen recordings using a Vision-Language Model to proactively recommend workflow improvements with a Language Model. The system accurately recognized 90.5% of user actions and was rated by users as more comprehensible, actionable, and effective for learning compared to a prompt-based AI assistant.
Key insights
- Visual data from screen recordings, when interpreted by advanced VLMs, can provide a robust, semantically rich representation of user actions and task context without needing application instrumentation.
- Proactive, behavior-grounded suggestions derived from observed user actions are more effective for learning and adoption than reactive, prompt-based AI assistance.
- Delivering workflow improvement advice post-task facilitates reflective learning and reduces cognitive load during active work, resembling effective "over-the-shoulder" mentorship.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.