Reproduce “TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zihan Xue, Po-Yi Lu, Serhii Honcharenko, Zih-Ching Chen, Hsuan-Tien Lin, Nanyun Peng, I-Hung Hsu, Kuan-Hao Huang
TwinICL introduces a paired benchmark for comparing text- and image-based in-context learning on the same examples, covering 38 shape-sequence tasks. Across six open-weight multimodal models, image inputs trail text inputs by an average of about 20 percentage points at 16 demonstrations; targeted support can largely close the gap on a smaller set of tasks, but explicit task instructions alone do not remove it.
Key insights
- At 16 shots, image accuracy remains below text accuracy for every evaluated open-weight model; the average gap is about 20 percentage points, but varies substantially by task family.
- A small modality gap can reflect low performance in both conditions: Relation tasks show a 2.1-point average gap, while some are near chance for text and images alike.
- Task induction is not the only source of the gap: a positive text–image difference persists when the rule is stated explicitly, and unlabeled demonstrations can have model-dependent effects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.