Reproduce “Recognition of Heat-Induced Food State Changes by Time-Series Use of Vision-Language Model for Cooking Robot” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Naoaki Kanazawa, Kento Kawaharazuka, Yoshiki Obinata, Kei Okada, Masayuki Inaba
Researchers from The University of Tokyo developed a VLM-based method for real-time recognition of heat-induced food state changes in cooking robots, employing time-series analysis of VLM outputs and prompt engineering. This approach demonstrated accurate recognition for vaporization, melting, and Maillard reactions, with up to 0.0s difference for Maillard reactions, improving autonomous culinary task execution by reacting to actual food transformations.
Key insights
- Prompts explicitly describing the visual manifestations of a state change (e.g., "Boiling and bubbling water") generally yield higher recognition confidence and performance compared to simpler descriptions.
- Counter-intuitively, a broader visual context (gazing at the "entire pot or pan") often leads to more accurate state change recognition than focusing only on the "contents" for VLMs.
- The VLM-based method shows strong generalizability to varying heating power conditions for specific state changes, indicating robustness against real-world variability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.