Reproduce “Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning” on OpenResearch

About this paper

Yu Zhu, Yongkang Li, Wenjie Zhu, Haoyi Jiang, Wenyu Liu, Wei Yang, Bin Li, Xinggang Wang

Researchers from Huazhong University of Science and Technology developed Food-R1, a unified multi-task food vision-language model, leveraging a new benchmark called CalorieBench-80K and a two-stage training pipeline that includes Chain-of-Thought instruction tuning and reinforcement learning. Food-R1 achieved a Mean Absolute Error of 42.56 kcal for calorie estimation, a Top-1 accuracy of 95.21% for food classification on Food-101, and an IoU of 78.50 for ingredient recognition on VireoFood-172.

Key insights

• Integrating Chain-of-Thought (CoT) annotations during instruction tuning helps models learn explicit, interpretable intermediate reasoning steps, improving accuracy and consistency in complex tasks like calorie estimation.
• Reinforcement learning, specifically GRPO with tailored reward functions, refines the model's reasoning trajectories beyond simple imitation, leading to more reliable outputs and enhanced overall performance.
• A unified multi-task VLM approach, leveraging a diverse set of food analysis tasks, enables knowledge transfer and shared visual semantics, resulting in more stable and accurate performance across a broad spectrum of food-related queries.

Summary by alphaXiv.