Reproduce “LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations” on OpenResearch

About this paper

Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, Tim Genewein

Google DeepMind introduces LMAct, an open-source benchmark for evaluating in-context imitation learning in large multimodal models using long demonstrations. The study reveals that state-of-the-art LMs largely fail to generalize from extensive multimodal demonstrations to achieve expert-level interactive decision-making, often showing performance plateaus or degradation despite context windows up to one million tokens.

Key insights

• A persistent "knowing-doing gap" was observed, as models rarely reached expert performance even with extensive in-context demonstrations, indicating a fundamental difficulty in generalizing learned patterns to effective action.
• Performance often plateaued or even degraded with increasing numbers of demonstrations across many tasks and models, suggesting that current LMs struggle to effectively utilize or filter information from dense, long contexts.
• While models demonstrated strong ability to accurately replay actions from a single provided demonstration, they struggled to translate this basic context retrieval into generalized interactive decision-making.

Summary by alphaXiv.