Reproduce “TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade
Researchers developed TN-Eval, a framework comprising a co-designed rubric and evaluation protocols, for assessing the quality of behavioral therapy notes. This framework demonstrated superior inter-annotator agreement compared to traditional Likert scales and revealed that while large language models can generate notes perceived as more complete and concise by therapists, they exhibit faithfulness challenges compared to human-written notes.
Key insights
- A fine-grained, rubric-based human evaluation protocol (TN_H-Eval) significantly improves inter-annotator agreement and provides more nuanced quality scores compared to traditional Likert scales.
- Large Language Models (LLMs) can generate behavioral therapy notes that are perceived by human therapists as superior in completeness and conciseness during blind evaluations.
- Despite apparent high quality, LLM-generated notes demonstrate lower faithfulness than human notes when rigorously evaluated with granular, source-attributing methods, highlighting persistent hallucination issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.