Reproduce “Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information” on OpenResearch

About this paper

Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, Willem Zuidema

Diagnostic classifiers were utilized to examine how Long Short-Term Memory (LSTM) models internally represent and track subject-verb agreement information, uncovering dynamic encoding patterns and identifying points of error. By applying insights from these classifiers, researchers successfully intervened in the LSTM's internal states, enhancing its verb agreement accuracy by an absolute 7.3 percentage points on original sentences.

Key insights

• Subject number information is most effectively encoded in the memory cell (c_t) and hidden activation (h_t) of the second LSTM layer, and this information is encoded dynamically rather than being statically maintained.
• Agreement errors made by the language model are associated with a significant loss or corruption of subject number information in internal states, particularly after agreement attractors, and can emerge early in processing.
• Diagnostic classifiers provide a causal understanding of internal representations, as interventions based on their predictions can directly improve the model's accuracy on linguistic tasks, validating the practical utility of interpretability methods.

Summary by alphaXiv.