Reproduce “HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Haris Khan, Miguel Altamirano Cabrera, Dmitrii Iarchuk, Yara Mahmoud, Daria Trinitatova, Issatay Tokmurziyev, Dzmitry Tsetserukou
Researchers at the Intelligent Space Robotics Laboratory at Skoltech developed HapticVLM, a system that uses Vision-Language Models to infer material properties and environmental context from visual input, enabling dynamic multimodal haptic feedback. The system achieved 86.7% accuracy in VLM-driven temperature estimation and an average of 84.7% accuracy in haptic pattern recognition by users.
Key insights
- Vision-Language Models (VLMs) can effectively infer ambient environmental conditions, such as temperature, from visual cues alone, eliminating the need for dedicated sensors.
- Material recognition using deep image encoders (like ConvNeXt) can accurately identify materials from visual input, enabling sensor-free inference of tactile properties.
- Combining visually-inferred material properties with VLM-estimated environmental context allows for dynamic and adaptable multimodal haptic feedback (vibrotactile and thermal).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.