Open-source AI matches top proprietary model in solving tough medical cases

open-source-aimedical-aillamagpt-4clinical-diagnosis

Abstraction: Open-source Llama 3.1 405B matches GPT-4 on clinical diagnostic reasoning

Key points:

NIH-funded Harvard Medical School study (published JAMA Health Forum, March 14 2025): Llama 3.1 405B matched GPT-4 performance on 92 diagnostically challenging NEJM clinical case scenarios.
Llama scored correct diagnosis in 70% of cases vs. 64% for GPT-4; ranked the correct diagnosis first 41% vs. 37% of the time; on 22 newer cases (post-training cutoff), Llama scored 73% correct.
Key advantage of open-source for healthcare: models can run on hospital's private computers, keeping patient data in-house rather than transmitted to external servers.
Open-source models can be fine-tuned on local clinical data for specific physician/patient populations; closed-source models are harder to customize.
Approximately 795,000 US patients die or suffer permanent disability from diagnostic errors annually (2023 report), underscoring the stakes.
First study to demonstrate open-source AI matching GPT-4 on such challenging clinical cases as assessed by physicians.

Connections: Meta · Openai · Harvard Medical School · Large Language Models · Medical AI · Open Source AI