LingxiDiagBench：一种用于比较中文精神科咨询与诊断领域大型语言模型性能的多智能体框架
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
摘要
精神障碍在全球范围内非常普遍，但精神科医生短缺以及基于访谈的诊断方式所具有的主观性，使得及时且准确的心理健康评估变得十分困难。人工智能辅助精神疾病诊断的发展受到缺乏能够同时提供真实患者模拟场景、经过临床医生验证的诊断标签以及支持动态多轮咨询的基准数据的限制。我们提出了LingxiDiagBench这一大规模多智能体基准数据集，该数据集用于评估大型语言模型在静态诊断推理和动态多轮精神疾病咨询方面的表现。其核心是LingxiDiag-16K，这是一个包含16,000条与电子病历一致的模拟咨询对话的数据集，旨在再现12种ICD-10精神疾病类别下的真实临床特征和诊断情况。通过针对最先进的语言模型进行的大量实验，我们得出了以下关键结论：(1)尽管语言模型在二元抑郁-焦虑分类任务上能达到很高准确率（高达92.3%），但在抑郁与焦虑同时存在的状况识别以及12种疾病的鉴别诊断方面，其性能则显著下降（分别为43.0%和28.5%）；(2)动态咨询方式的评估结果通常低于静态评估结果，这表明无效的信息收集策略会严重影响后续的诊断过程；(3)由语言模型作为评判者所评估的咨询质量与诊断准确性之间的相关性仅为中等水平，这说明良好的提问方式本身并不能确保正确的诊断决策。我们将在https://github.com/Lingxi-mental-health/LingxiDiagBench上发布LingxiDiag-16K及完整的评估框架，以支持可重复性的研究工作。
English Abstract
Mental disorders are highly prevalent worldwide, but the shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment. Progress in AI-assisted psychiatric diagnosis is constrained by the absence of benchmarks that simultaneously provide realistic patient simulation, clinician-verified diagnostic labels, and support for dynamic multi-turn consultation. We present LingxiDiagBench, a large-scale multi-agent benchmark that evaluates LLMs on both static diagnostic inference and dynamic multi-turn psychiatric consultation in Chinese. At its core is LingxiDiag-16K, a dataset of 16,000 EMR-aligned synthetic consultation dialogues designed to reproduce real clinical demographic and diagnostic distributions across 12 ICD-10 psychiatric categories. Through extensive experiments across state-of-the-art LLMs, we establish key findings: (1) although LLMs achieve high accuracy on binary depression--anxiety classification (up to 92.3%), performance deteriorates substantially for depression--anxiety comorbidity recognition (43.0%) and 12-way differential diagnosis (28.5%); (2) dynamic consultation often underperforms static evaluation, indicating that ineffective information-gathering strategies significantly impair downstream diagnostic reasoning; (3) consultation quality assessed by LLM-as-a-Judge shows only moderate correlation with diagnostic accuracy, suggesting that well-structured questioning alone does not ensure correct diagnostic decisions. We release LingxiDiag-16K and the full evaluation framework to support reproducible research at https://github.com/Lingxi-mental-health/LingxiDiagBench.