Reproduce “DevBench: A multimodal developmental benchmark for language learning” on OpenResearch

About this paper

Alvin Wei Ming Tan, Sunny Yu, Bria Long, Wanjing Anya Ma, Tonya Murray, Rebecca D. Silverman, Jason D. Yeatman, Michael C. Frank

Stanford University researchers developed DEVBENCH, a multimodal developmental benchmark that facilitates systematic comparison of vision-language models' language learning trajectories and response patterns with human children and adults. It comprises seven tasks covering lexical, syntactic, and semantic abilities, leveraging item-level human behavioral data and metrics like softmax-optimised KL divergence and Representational Similarity Analysis to evaluate human-likeness.

Key insights

• Model-human similarity correlated most strongly with task accuracy across ages and tasks, suggesting performance level is a key factor in human-like response patterns.
• Analyzing model training trajectories revealed that earlier checkpoints of OpenCLIP-H aligned with younger children's performance, while later checkpoints resembled older children and adults, mirroring developmental progression for certain tasks.
• Item-level analysis indicated models struggle with ambiguity, polysemy, and context-dependent interpretations (e.g., "horn," "net") where humans typically leverage pragmatic reasoning.

Summary by alphaXiv.