Reproduce “On the robustness of modeling grounded word learning through a child's egocentric input” on OpenResearch

About this paper

Wai Keen Vong, Brenden M. Lake

Researchers from New York University demonstrated that multimodal neural networks can learn and generalize word-referent mappings from large, automatically transcribed egocentric video datasets of multiple children. The models showed robust performance across children and modest generalization to novel visual domains, with learning outcomes correlating with the quantity of high-quality vision-language aligned examples in the input.

Key insights

• Automated speech transcription is sufficiently robust for grounded word learning tasks, yielding classification performance comparable to models trained on manually transcribed data.
• Word-referent mappings are robustly learnable across different children's egocentric inputs, with individual differences in learning outcomes correlating positively with the absolute number of high-quality vision-language aligned examples.
• Learned word embeddings demonstrate generalizability across varying videos from the same child, to the visual contexts of other children, and, to a more limited extent, to novel image domains, with performance decreasing as domain shift increases.

Summary by alphaXiv.