ChatGPT Makes OK Clinical Decisions—Usually
ai-healthcarechatgptclinical-decision-supportllm-evaluation
Abstraction: Study finds ChatGPT 72% accurate across clinical decision tasks using fictional vignettes
Key points:
- Study used all 36 Merck Manual clinical vignettes; ChatGPT scored 72% overall across differential diagnosis, test ordering, final diagnosis, and treatment recommendation
- Best performance: final diagnosis after testing results (77%); worst: differential diagnosis from initial presentation only (60%)
- No detectable bias by patient age or gender; however, underlying medical data biases (e.g., pulse oximetry, female heart attack symptoms) could still propagate
- Published in Journal of Medical Internet Research, August 2023; conducted by Mass General Brigham researchers
- Key limitation: small sample, fictional patients, and opaque training data make it difficult to generalize or control outputs
- Experts agree chatbots could assist but never replace physicians; clinical trial processes would be needed before real-patient use
Connections: Chatgpt · Openai · AI In Healthcare · Clinical Decision Support · Large Language Models