Reproduce “Personalized Digital Semantic Communication for Image Transmission with Vision-Language Models” on OpenResearch

About this paper

Nan Li, Li Zhou, Haijun Wang, Jun Xiong, Haitao Zhao, Jibo Wei

A personalized digital semantic communication (PDSC) framework leverages vision-language models and latent diffusion models to transmit images, reducing bandwidth while reconstructing visual content aligned with individual receiver preferences. The system demonstrates superior performance in both user-preference alignment and source-semantic consistency compared to state-of-the-art baselines, particularly under low signal-to-noise ratio and constrained bit rates.

Key insights

• Explicitly incorporating receiver-dependent semantics into the communication pipeline significantly enhances both transmission efficiency and the quality of reconstructed images in terms of user alignment.
• The combination of advanced vision-language models for semantic token extraction and latent diffusion models for generative reconstruction provides a powerful approach for efficient and personalized image transmission.
• Formulating a personalized rate-distortion problem with a composite distortion metric allows for principled optimization, enabling the system to intelligently allocate bit budgets to preserve both source meaning and individual preferences.

Summary by alphaXiv.