Reproduce “PUMGPT: A Large Vision-Language Model for Product Understanding” on OpenResearch

About this paper

Wei Xue, Zongyi Guo, Baoliang Cui, Zheng Xing, Xiaoyi Zeng, Xiufei Wang, Shuhui Wu, Weiming Lu

Researchers from Zhejiang University and Alibaba Group developed PUMGPT, a large vision-language model specialized for e-commerce product understanding, which significantly outperforms general-purpose LVLMs like GPT-4V on tasks such as attribute inference and caption generation. The model achieves this by leveraging a multi-expert hallucination detection framework to curate a high-quality, domain-specific dataset from over one million AliExpress products.

Key insights

• Domain-specific fine-tuning and meticulous data curation, especially hallucination filtering, are crucial for large vision-language models to achieve high precision and practical utility in specialized applications like e-commerce.
• A multi-expert collaborative framework can effectively identify and filter out 'hallucinated' or un-inferable attributes in noisy real-world datasets, significantly enhancing data quality and model reliability.
• A single, unified LVLM can effectively handle multiple inter-related product understanding tasks, offering a more efficient and adaptable solution compared to deploying separate task-specific models.

Summary by alphaXiv.