Reproduce “LM4LV: A Frozen Large Language Model for Low-level Vision Tasks” on OpenResearch

About this paper

Boyang Zheng, Jinjin Gu, Shijun Li, Chao Dong

This research from Shanghai AI Laboratory and partner institutions demonstrates that a frozen Large Language Model can effectively process and generate raw, pixel-level visual features for low-level vision tasks. By integrating a fine-tuned Masked Autoencoder with minimal adapter layers, LM4LV achieved significant quantitative improvements, such as an average 3.96dB PSNR increase across restoration tasks, without relying on any multi-modal training data or priors.

Key insights

• Frozen Large Language Models (LLMs) possess an inherent, non-trivial capacity to process and generate raw, pixel-level visual features for low-level vision tasks, challenging the view that they are only capable of high-level semantic understanding.
• The success of interfacing LLMs with vision tasks hinges on using a vision module that preserves comprehensive pixel-level information (e.g., a fine-tuned Masked Autoencoder) and leveraging the LLM's auto-regressive generation capability.
• Extensive text pre-training endows LLMs with generalized understanding that proves crucial for solving low-level vision tasks, indicating that their acquired intelligence extends beyond linguistic processing to other structured data.

Summary by alphaXiv.