Reviewed by Danielle Ellis, B.Sc.Sep 27 2024
In a recent study published in the Joint Commission Journal on Quality and Patient Safety, researchers from Boston University tested GPT-4, an advanced publicly available generative AI model, to evaluate its ability to answer questions across five key areas of patient safety from the 50-question self-assessment of the Certified Professional in Patient Safety (CPPS) exam—a standardized multiple-choice certification for patient safety professionals. GPT-4 achieved an 88% accuracy rate, showcasing a strong level of performance in this domain.
Using hundreds of millions, even billions, of data points, generative artificial intelligence (genAI) trains itself to generate creative and lifelike outputs that can imitate material created by humans. It can be used to speed up engineering design, create audio and video content, and offer tailored recommendations to online customers.
Potential applications of genAI in healthcare include improving imaging technology, forecasting a patient's disease trajectory, and developing novel vaccinations.
While other studies have looked at genAI's performance on exams from different healthcare specialties over the past year, ours is the first robust test of its proficiency specifically in patient safety.
Nicholas Cordella, Study Corresponding Author and Assistant Professor, Boston University
James Moses, MD, MPH, co-author of the report, was the school's Associate Professor of Pediatrics before taking a position as Chief of Quality, Safety, and Patient Experience at Corewell Health in Michigan.
Without any extra training or medical fine-tuning, the researchers fed the GPT-4 model questions from the CPPS self-assessment exam. Subsequently, they assessed the model's effectiveness in other exam categories.
The fields of patient safety and solutions, measuring and improving performance, and systems thinking and design/human factors were determined to be where GPT-4 performed well. The researchers identified areas where patient safety professionals may start exploring the practical capabilities and limitations of AI based on the quality of those results.
Our findings suggest that AI could help doctors better recognize, address, and prevent mistakes in hospitals and clinics. While more research is needed to fully understand what current AI can do in patient safety, this study shows that AI has some potential to improve healthcare by assisting clinicians in addressing preventable harms.
Nicholas Cordella, Study Corresponding Author and Assistant Professor, Boston University
Cordella is also the Medical Director for Quality and Patient Safety at Boston Medical Center.
He thought that applying AI could help enhance patient safety protocols and more effectively address the unsolvable issue of medical errors, which are thought to be the cause of almost 400,000 fatalities annually.
According to Cordella, the study supports the more general notion that AI can improve the work of experts, such as physicians. Similar to how other knowledge workers are utilizing AI to enhance their performance, clinicians may be able to increase the security and effectiveness of healthcare by leveraging it to support their activities.
The study also pointed out the shortcomings of existing AI technology and advised users to be on the lookout for bias, false confidence, faked data, or hallucinations in responses from huge language models (such as the GPT-4).
Our findings suggest that AI has the potential to significantly enhance patient safety, marking an enabling step towards leveraging this technology to reduce preventable harms and achieve better healthcare outcomes. However, it is important to recognize this as an initial step, and must rigorously test and refine AI applications to truly benefit patient care.
Nicholas Cordella, Study Corresponding Author and Assistant Professor, Boston University
Journal Reference:
Cordella, N., et al. (2024) Artificial Intelligence and the Practice of Patient Safety: GPT-4 Performance on a Standardized Test of Safety Knowledge. The Joint Commission Journal on Quality and Patient Safety. doi.org/10.1016/j.jcjq.2024.05.007.