Reproduce “Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinsong Chen
A novel psychometric method is introduced for textual data analysis, leveraging Large Language Model contextual embeddings by conceptualizing documents as individuals and words as items. This approach successfully uncovers latent knowledge dimensions, extracting meaningful and interpretable semantic constructs from corpora.
Key insights
- The conceptual framework of treating documents as "individuals" and words as "items" enables the direct application of psychometric modeling to contextual text embeddings.
- Contextual scores, derived from LLM embeddings, exhibit robust psychometric properties suitable for factor analysis, including moderate non-normality and stable distributions across data splits.
- Bifactor analysis provides a superior interpretation of latent knowledge dimensions, clearly separating broad general factors (e.g., "Astronomy," "Biology") from specific minor factors, with general factors defined by a large number of keywords.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.