Development and evaluation of a large language model-based automated nerve conduction study interpretation system

초록

Background and aims: Nerve Conduction Study (NCS) interpretation requires significant expertise and time, with variability among practitioners. We aimed to develop and evaluate an AI-based NCS interpretation model utilizing Large Language Model (LLM) technology and compare its performance to both commercial LLMs and human experts. Methods: We collected NCS data from patients at Korea University Anam Hospital (2010–2024), excluding cases with missing data or test errors. Raw data was converted into structured text representations using rule-based coding. We fine-tuned GPT-4o (NCS-LLM) on a training set (n = 1335) and evaluated it against six commercial LLMs (GPT-4.5, GPT-4o, o3-mini, Claude 3.7, Claude 3.7-thinking, Llama 3.3 70B) and human experts (trainees and junior neurophysiologists) on a test set (n = 66). Board-certified neurophysiologists evaluated the interpretations using a 5-point Likert scale for accuracy and clinical rationale. Generation efficiency was also measured. Results: The fine-tuned NCS-LLM outperformed all commercial LLMs in both accuracy and efficiency. Compared to human experts, the model performed better than trainees and matched junior neurophysiologists in interpretation accuracy while demonstrating significantly superior efficiency in generating interpretations. Conclusions: This study demonstrates that a fine-tuned LLM can interpret NCS data with accuracy comparable to human experts while offering substantial efficiency gains. As the first application of LLMs to NCS interpretation, our approach offers a promising foundation for developing multi-modal diagnostic models incorporating additional clinical information and electromyography data to enhance peripheral neuropathy diagnosis.