Reproduce “Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Divyam Madaan, Varshan Muhunthan, Kyunghyun Cho, Sumit Chopra
The authors test how visual-question-answering models use images and questions across 23 multiple-choice benchmarks by shuffling inputs under four conditions and comparing accuracy. Using a majority-vote ensemble of three Cambrian-1 models, they find modality reliance varies across benchmarks and categories; on MMBench, image-only accuracy exceeds the random baseline by 41%. This measures broken-alignment performance, not its cause or other tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.