Reproduce “RNNs Implicitly Implement Tensor Product Representations” on OpenResearch

About this paper

R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky

A new Tensor Product Decomposition Network (TPDN) was developed to analyze if Recurrent Neural Networks (RNNs) implicitly implement compositional structures. The study found that RNNs learn Tensor Product Representations (TPRs) when trained on structure-sensitive tasks, but current natural language models' representations often prioritize bag-of-words information over robust structural encoding.

Key insights

• Recurrent Neural Networks (RNNs) can implicitly learn and implement Tensor Product Representations (TPRs) when trained on tasks requiring structural sensitivity.
• The specific type of compositional structure learned by an RNN is heavily influenced by the decoder architecture and the demands of the training task.
• While state-of-the-art natural language sentence encoders exhibit some structural sensitivity, their representations are largely dominated by non-structural (bag-of-words) information, suggesting current training objectives may not fully incentivize robust compositional learning.

Summary by alphaXiv.