Reproduce “MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John Mendonça, Daniel Tschernutter, José Pombal, Areti Vassilopoulos, Catarina Botelho, Ignacio Sarasua, Maya D'Eon, Nuno Guerreiro and 1 more
MindEval2 evaluates everyday, non-crisis AI support using simulated patients who return across sessions, and scores specific counselor-turn issues rather than whole conversations. Its MHQR rubric weights issue severity, while shared memory lets patients and counselors use prior-session context. Over-exploring, under-guiding, and bad advice account for 60–88% of tested counselors’ scores; this is not a clinical-outcome or safety test.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.