Can ChatGPT effectively generate abstracts in orthopedic surgery? A comparative analysis between human-written and ChatGPT-generated scientific abstracts

초록

Background Despite growing interest in using Chat-based Generative Pre-trained Transformer (ChatGPT) for academic writing, limited evidence exists regarding its ability to generate abstracts that are structurally compliant and ethically acceptable in orthopedic surgery. Objective To assess the performance of ChatGPT-generated abstracts using only article titles from recent publications in major orthopedic journals. Methods We extracted 90 human-written abstracts from three leading orthopedic journals and used each title to generate abstracts with ChatGPT-3.5 and ChatGPT-4.0. A total of 180 AI-generated abstracts were created using a standardized prompt. Each abstract was evaluated for format compliance, adherence to word limit, word count, consistency in study design, sample size correlation, and conclusion relevance. Plagiarism and AI detectability were assessed. Four orthopedic surgeons independently reviewed a subset of abstracts to identify their source. Results GPT-4.0 achieved perfect compliance with journal format and word count, while GPT-3.5 met these criteria in 34.4% (31 of 90) and 86.7% (78 of 90) of cases, respectively (P < .001). However, only half of abstracts presented fully relevant conclusions. Plagiarism was flagged in 45% to 70% of cases across both detection programs. AI detection scores were significantly higher in GPT-generated abstracts than for human-written ones (P < .001). Human reviewers showed limited ability to distinguish between human and AI-generated abstracts, with minimal inter-rater agreement (Cohen's kappa = 0.25). Conclusion Although ChatGPT, particularly GPT 4.0, can generate abstracts that meet structural requirements and reproduce surface-level elements of academic style, significant limitations remain in content accuracy, originality, and ethical considerations.

키워드