Reproduce “A Large-Scale Benchmark for Food Image Segmentation” on OpenResearch

About this paper

Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven C.H. Hoi, Qianru Sun

This work introduces FoodSeg103 and FoodSeg154, large-scale datasets featuring fine-grained, pixel-wise segmentation masks for up to 154 food ingredient categories. It also proposes the Recipe Learning Module (ReLeM), a multi-modality pre-training approach that leverages semantic knowledge from cooking recipes to consistently improve segmentation model performance across various architectures.

Key insights

• Fine-grained food ingredient segmentation is a significantly more challenging task compared to conventional scene segmentation, exhibiting a substantial performance gap even for state-of-the-art models.
• Leveraging multi-modal information from recipes (textual ingredient lists and instructions) effectively reduces the high intra-class variance of food ingredients, leading to more robust visual feature learning.
• The LSTM-based text encoder within ReLeM generally outperformed transformer-based encoders for encoding recipe semantic information, suggesting its effectiveness for this specific task.

Summary by alphaXiv.