Reproduce “Otter: A Multi-Modal Model with In-Context Instruction Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Ziwei Liu

Otter is a multi-modal model that introduces in-context instruction tuning, leveraging both visual and textual examples to significantly enhance instruction following in Large Multimodal Models (LMMs). It achieves state-of-the-art or highly competitive performance on a wide range of image and video understanding benchmarks, notably improving video QA scores by substantial margins (e.g., 77.9 on MSVD-QA vs. 64.1 for previous best).

Key insights

  • Explicitly training LMMs with multimodal in-context examples significantly enhances model convergence, generalization capabilities, and robustness compared to tuning without such context.
  • Task descriptions, when prepended as prefixes to instructions during training, substantially improve model performance on specific benchmarks by guiding the model's focus and response format.
  • Multimodal in-context instruction tuning not only improves instruction following but also preserves and amplifies the model's inherent in-context learning (ICL) capabilities, which can sometimes degrade with standard instruction tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.