Reproduce “Large Language Models as Developmental Raters: Convergent Triangulation, Blinded Reliability, and a Measured Feature-Ceiling Artifact” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Christa Diaz
Researchers at Christa Diaz Consulting LLC developed an auditable, LLM-based system to rate developmental stages from naturalistic text transcripts at high temporal density. The system demonstrated strong self-agreement with an ICC(2,1) of 0.786, and revealed a "feature-ceiling artifact" where the LLM consistently underutilized the top of its assigned scoring scale, requiring external anchor validation.
Key insights
- LLM-based systems can achieve strong self-consistency (ICC(2,1) of 0.786) and internal coherence (Cronbach's alpha of 0.953) on naturalistic text when confined to bounded feature-scoring tasks, enabling high-density longitudinal data collection.
- The 'feature-ceiling artifact,' where an LLM systematically underutilizes the upper end of its scoring scale, can be invisible to internal psychometric statistics and only revealed by comparison to external, held-out anchors.
- External validation against independent anchor points is indispensable for computational psychometric instruments using LLMs, highlighting a critical discipline for building trustworthy rater systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.