Detection of Mental Health Symptoms From Chatbot Conversations Using a Language Model

초록

Introduction/Aims: Early identification of clinically relevant mental health symptoms is essential for timely intervention, yet scalable screening approaches remain limited. With the increasing use of conversational AI, passive analysis of user-generated dialogue may offer a novel opportunity for real-time symptom detection. This study aimed to develop and evaluate a language model for detecting mental health symptoms from chatbot-based conversations. Method: We analyzed conversational data from 152 participants who interacted with a conversational AI agent (“Luda Lee”) over a 4-week period. After data augmentation, 1176 conversation samples were included. Mental health symptoms were assessed using validated instruments, including the PHQ-9, GAD-7, PSS-10, LSAS, and PHQ-GAD. A pre-trained BERT model was fine-tuned on 70% of the dataset, and performance was evaluated on a held-out test set. Results: The model demonstrated strong performance across symptom domains, with F1 scores of 95.7% (PHQ-9), 97.1% (GAD-7), 98.0% (PSS-10), 82.7% (LSAS), and 95.7% (PHQ-GAD). These findings indicate that conversational language patterns can reliably reflect clinically relevant mental health symptoms. Conclusion: Conversational AI–based analysis may provide a scalable and non-intrusive approach for early detection of mental health symptoms. This approach has potential as a clinical decision-support tool and may help facilitate early intervention in real-world settings.