Reproduce “LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, Jian Tang
The LLaVA-Phi model, developed by researchers from Midea Group and East China Normal University, presents an efficient multi-modal assistant built upon the 2.7 billion parameter Phi-2 language model. It delivers competitive visual comprehension and reasoning capabilities across standard benchmarks, often exceeding the performance of models three times its size, making it suitable for resource-constrained applications.
Key insights
- Small Language Models (SLMs) like Phi-2, when paired with robust visual encoders and trained on high-quality multi-modal data, can achieve performance levels comparable to, or surpassing, much larger Vision-Language Models.
- The specific pre-training of Phi-2 on code generation and mathematical corpora translates into enhanced multi-modal capabilities, particularly evident in scientific reasoning and accurate code generation from images.
- A refined two-stage multi-modal training methodology, where the projector and LLM are progressively fine-tuned, is effective in creating highly efficient multi-modal assistants suitable for resource-constrained deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.