Reproduce “Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities” on OpenResearch

About this paper

Kaiwen Cai, Zhekai Duan, Gaowen Liu, Charles Fleming, Chris Xiaoxuan Lu

EdgeVL introduces a two-stage framework to adapt large Vision-Language models like CLIP for efficient open-vocabulary classification on resource-constrained edge devices, accommodating diverse visual modalities without manual annotations. This approach achieves significant model size reductions (up to 93-fold) and latency improvements (around 50%) while boosting accuracy on non-RGB images by up to 13.9% on benchmarks like EuroSAT.

Key insights

• Automated dataset curation, leveraging large language models for label superset generation and teacher confidence scores for data filtering, effectively addresses label scarcity in edge environments.
• A unified image encoder with shared weights for both RGB and non-RGB inputs simplifies the student model architecture and enables effective cross-modal knowledge transfer.
• Combining Quantization-Aware Training (QAT) with contrastive learning is crucial for maintaining and improving feature discriminability in severely compressed, low-bit VL models.

Summary by alphaXiv.