Reproduce “O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit, Aditay Tripathi, Anirban Chakraborty
Researchers from the Indian Institute of Science developed O3SLM, an open-source, sketch-language model, alongside the large-scale SketchVCL dataset to equip Large Vision-Language Models with robust sketch comprehension. This approach significantly enhances performance across sketch-based tasks, achieving an average accuracy of 31.7% on COCO for counting, surpassing GPT-4o's 16.4%, and an Acc@1 of 65.0% for Sketch-Based Image Retrieval on Sketchy, compared to LLaVA-1.5-7B's 11.0%.
Key insights
- The availability of a large-scale, instruction-aligned multi-modal dataset like SketchVCL is fundamental for teaching LVLMs to understand abstract sketches.
- A two-stage training methodology, starting with broad sketch alignment pretraining and progressing to task-specific instruction tuning, effectively integrates sketch comprehension.
- Tuning the multimodal projector, rather than freezing it, significantly improves the integration of sketch and image features, leading to higher performance in detection tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.