Reproduce “Discovering Latent Concepts Learned in BERT” on OpenResearch

About this paper

Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, Hassan Sajjad

Researchers from Qatar Computing Research Institute and Stevens Institute of Technology introduced an unsupervised methodology to discover latent concepts within BERT's contextualized representations. Their approach revealed novel, multi-faceted concepts, identified instances of learned bias, and demonstrated the layer-wise evolution of these concepts, further contributing the BERT ConceptNet dataset for future interpretability research.

Key insights

• BERT learns diverse, often multi-faceted latent concepts that extend beyond pre-defined linguistic categories, indicating the model's ability to infer novel and compositional meanings.
• The unsupervised discovery process revealed that BERT's internal representations encode biases, such as gender stereotypes and demographic associations, inherited from its training data.
• Concept learning within BERT exhibits a hierarchical progression, with lower layers focusing on shallow lexical and initial meaning, while higher layers integrate context to form more complex linguistic and semantic representations.

Summary by alphaXiv.