Human-written medical histories rated higher than ChatGPT in paediatric orthopaedics
Photo by Andrew Neel / Unsplash
Key Takeaway
Consider that human-written medical histories remain superior to ChatGPT-generated ones in paediatric orthopaedics, especially for accuracy and consistency.
This randomized, blinded comparative study evaluated the quality of medical history summaries created by humans versus ChatGPT in a paediatric orthopaedic practice. Twenty consecutive paediatric patients (mean age 14.2 ± 2.3 years; 11 males, 9 females) were included. Each patient had two summaries: one human-created and one ChatGPT-generated, both based on the same clinical encounter.
Primary outcome was overall rating on a 6-point Likert scale. Human summaries scored significantly higher (5.2 ± 0.8) than ChatGPT summaries (4.5 ± 0.8), with a large effect size (Cohen's d = 0.80, p < 0.001). Secondary outcomes including temporal consistency, spatial consistency, accident description, and overall impression also significantly favored human documentation (all p < 0.001). For example, ChatGPT had errors in accident description in 21 of 60 evaluations (35%) and in temporal consistency in 14 of 60 evaluations (23%). No significant differences were found for writing style or documentation of previous interventions.
Safety and tolerability were not reported. A key limitation was moderate inter-rater reliability (ICC = 0.64). The study suggests that current large language models are not ready to replace human medical documentation in paediatric orthopaedic practice without careful oversight, supporting hybrid workflows where AI assists but does not replace human clinical judgement.
View Original Abstract ↓
PURPOSE: This study evaluated the quality of ChatGPT-generated medical history summaries compared to human-created documentation in a paediatric orthopaedic practice setting.
METHODS: A prospective, randomised, blinded comparative study was conducted involving 20 consecutive paediatric patients (mean age 14.2 ± 2.3 years; 11 males, 9 females) presenting with knee problems. Audio recordings of medical consultations were transcribed and processed by ChatGPT-4o (OpenAI) using standardised prompts. Three independent orthopaedic specialists evaluated both human-generated and AI-generated summaries using eight quality criteria: temporal consistency, spatial consistency, accident description, symptom accuracy, symptom specificity, previous interventions, writing style and overall impression. Each criterion was scored on a 6-point Likert scale.
RESULTS: Human-created summaries received significantly higher overall ratings (5.2 ± 0.8) compared to ChatGPT-generated summaries (4.5 ± 0.8, p < 0.001, Cohen's d = 0.80). After Bonferroni correction for multiple comparisons, statistically significant differences favouring human documentation were confirmed in four of eight criteria: temporal consistency (p < 0.001), spatial consistency (p < 0.001), accident description (p < 0.001) and overall impression (p < 0.001). No significant differences were observed for writing style and documentation of previous interventions. Inter-rater reliability was moderate (ICC = 0.64). ChatGPT demonstrated frequent temporal inconsistencies (14 of 60 evaluations, 23%) and omission of relevant accident details (21 of 60 evaluations, 35%).
CONCLUSION: While AI-generated summaries showed acceptable stylistic quality, human documentation significantly outperformed ChatGPT in critical clinical dimensions, including temporal consistency and accuracy of complex orthopaedic presentations. Current large language models are not ready to replace human medical documentation in paediatric orthopaedic practice without careful oversight. The findings support the implementation of hybrid workflows where AI assists but does not replace human clinical judgement.
LEVEL OF EVIDENCE: Level I.
VBQ and Hounsfield Units Predict Subsequent Fractures After Vertebral Augmentation
Vertebral Bone Quality Scores Predict Fracture Risk After Vertebral Augmentation
Related in Orthopedics & Sports Medicine
Calcium sulfate paste graft with locking plate for proximal humeral fractures: no functional benefit
Adding calcium paste to bone plates reduced complications in older patients with broken shoulders.
This randomized clinical trial in 70 older patients with proximal humeral fractures compared osteosynthesis using a locking plate with calci…
Adding calcium paste to bone plates reduced total complications in older patients with broken shoulders, while other recovery measures staye…
Intramedullary fixation with dual integrated lag screw shows no difference in failure rates compared to single lag screw fixation.
Static Locking Screws Work Just As Well As Dynamic Ones For Hip Fractures
This multicenter randomized controlled trial enrolled 477 patients older than 60 years with intertrochanteric femur fractures. The study com…
New data shows that locking the top screw in place does not raise the risk of hip implant failure for older patients.
Contracting retraining schedule improves task speed compared to expanding schedule in novice orthopedic residents
Practicing more before surgery helps doctors keep skills sharp
This randomized controlled trial compared three retraining schedules for 27 orthopedic residents without prior arthroscopic experience. The …
Practicing more often right before a break helps surgeons remember their skills better than spreading out their training time
VBQ and Hounsfield Units Predict Subsequent Fractures After Vertebral Augmentation
Vertebral Bone Quality Scores Predict Fracture Risk After Vertebral Augmentation
This meta-analysis of 7364 patients found that higher VBQ scores and lower HU values are significantly associated with subsequent fractures …
A large analysis of 7,364 patients shows Vertebral Bone Quality scores better predict future fractures after vertebral augmentation than sta…
From Other Specialties
Educational video improves spirometry success in preschool children
Short Video Boosts Spirometry Success in Preschoolers
This randomized controlled trial in 132 preschool children found that watching a short educational video before spirometry significantly imp…
A short educational video helped 84% of preschoolers complete a lung test successfully, compared to 66% with standard instructions.
Mental health information and diagnosis reduced guilt ratings in mock criminal trial participants
More mental health details in court lowered guilt ratings for those with higher knowledge
This randomized controlled trial involved 243 mock criminal trial participants. The study compared guilt ratings across conditions varying i…
Adding mental health details and diagnoses to a case reduced guilt ratings, especially for people who understand these conditions better.
Virtual Reality methods improved reproductive health and family planning attitudes in women compared to face-to-face training
Virtual reality training boosts family planning attitudes in women
This randomized controlled trial enrolled 120 patients across three groups to evaluate Virtual Reality methods for reproductive health and f…
A new trial shows virtual reality methods significantly improve reproductive health attitudes compared to standard face-to-face training for…
Disaster education boosts preparedness in Turkish women, randomized trial finds
Disaster education boosts preparedness in women in western Turkey
This randomized controlled trial in 120 women in western Turkey found that a structured disaster awareness program significantly improved di…
A four-week course on disaster types and first aid significantly improved awareness and preparedness scores in women compared to a nutrition…