Reproduce “Vector-ICL: In-context Learning with Continuous Vector Representations” on OpenResearch

About this paper

Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, Jianfeng Gao

The Vector-ICL method from UC San Diego and Microsoft Research enables large language models to perform in-context learning on continuous vector representations from various data modalities. This approach allows LLMs to handle non-textual inputs like fMRI data, time-series, and graphs, demonstrating improved precision in numerical reasoning and outperforming traditional baselines across diverse tasks.

Key insights

• LLMs can effectively interpret and learn from abstract continuous vector representations, extending their inherent generalization capabilities beyond discrete textual inputs.
• A modular framework using lightweight, trainable projectors connected to frozen LLMs and black-box encoders offers an efficient way to integrate diverse data modalities without re-training the core LLM.
• Continuous vector representations provide a more robust mechanism for LLMs to handle precise numerical information, overcoming the limitations imposed by discrete tokenization schemes.

Summary by alphaXiv.