Reproduce “SkillSight: Efficient First-Person Skill Assessment with Gaze” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chi Hsuan Wu, Kumar Ashutosh, Kristen Grauman
SkillSight, a two-stage multimodal framework for first-person skill assessment from smart glasses, enables accurate skill level classification by jointly modeling egocentric video and gaze. Its power-efficient gaze-only student model reduces energy consumption by up to 73 times compared to video-based methods, while maintaining high performance across diverse activities.
Key insights
- Gaze patterns serve as highly discriminative cues for skill assessment across diverse and dynamic real-world activities, extending gaze-based analysis beyond static, hand-centric tasks.
- Knowledge distillation effectively transfers visual information from a multimodal teacher to a gaze-only student, enabling accurate skill assessment without direct video input and drastically reducing power consumption.
- A comprehensive approach combining visual attention, explicitly modeled attended object sequences, and 3D gaze dynamics provides a robust understanding of skill, aligning with psychological theories on expertise and attention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.