Reproduce “TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junlong Jia, Ying Hu, Xi Weng, Yiming Shi, Miao Li, Xingjian Zhang, Baichuan Zhou, Ziyu Liu and 3 more
TinyLLaVA Factory offers an open-source, modular codebase for creating, training, and evaluating small-scale Large Multimodal Models (LMMs) with affordable computational resources. The framework reliably reproduces TinyLLaVA models, demonstrating slightly superior performance compared to original results on various image question-answering and comprehensive LMM benchmarks.
Key insights
- The modular software design, employing a factory pattern, significantly simplifies the complexity of LMM development, allowing for easy interchangeability of components like LLMs, vision towers, and training recipes.
- Focusing on 'small-scale LMMs' (450M to 2.7B parameters) makes advanced multimodal AI research accessible to a wider range of researchers with affordable computational resources.
- Standardized training and evaluation pipelines, combined with detailed documentation and extensive integration tests, address the crucial issue of reproducibility in deep learning research.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.