AI-Powered Wearable Camera Detects Medication Errors

Researchers from the University of Washington Medicine have developed the first wearable camera system that uses artificial intelligence (AI) to detect potential medication delivery errors. Their findings were published in npj Digital Medicine on October 22, 2024.

According to the results of the test, the video system demonstrated high proficiency in identifying medications being drawn in busy clinical settings. The AI’s sensitivity and specificity for detecting vial-swap errors were 99.6 % and 98.8 %, respectively.

The AI-powered camera system was developed and tested at the University of Washington. Co-lead author Dr. Kelly Michaelsen, an assistant professor of anesthesiology and pain medicine at the University of Washington School of Medicine, noted that the system could become a crucial safety tool, particularly in operating rooms, intensive care units, and emergency medicine settings.

The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful. One can hope for a 100 % performance, but even humans cannot achieve that. In a survey of more than 100 anesthesia providers, the majority desired the system to be more than 95 % accurate, which is a goal we achieved.

Dr. Kelly Michaelsen, Study Co-Lead Author and Assistant Professor, Anesthesiology and Pain Medicine, University of Washington School of Medicine

Drug administration errors are the most common cause of serious medical mistakes in intensive care and the most frequently reported critical incidents in anesthesia. It is estimated that errors occur in 5 % to 10 % of the drugs administered. Each year, 1.2 million patients are believed to experience adverse events related to injectable drugs, resulting in an estimated cost of $5.1 billion.

Syringe and vial-swap errors are most likely to occur during intravenous injections, when a physician transfers medication from a vial to a syringe for the patient. Substitution errors—such as selecting the wrong vial or mislabeling a syringe—account for 20 % of these mistakes. Another 20 % of errors occur when the medication is correctly labeled but administered incorrectly.

To reduce such errors, safety measures like barcode systems are in place to quickly read and verify the contents of a vial. However, because this step adds to their workflow, practitioners may occasionally skip it in high-pressure situations.

The researchers aimed to develop a deep learning model, integrated with a GoPro camera, capable of identifying the contents of syringes and cylindrical vials and issuing a warning before the patient receives the medication.

Training the model took several months. The team collected 4K video footage of 418 drug draws performed by 13 anesthesiologists in operating rooms with varying lighting and setups. The clinicians were filmed handling vials and syringes of specific drugs, and these video clips were later labeled with the contents of the vials and syringes to train the model.

Rather than relying on reading the text on each vial, the system uses additional visual cues—such as the size and shape of the vial and syringe, the color of the vial cap, and the size of the label print—to identify the medication.

It was particularly challenging, because the person in the OR is holding a syringe and a vial, and you don’t see either of those objects completely. Some letters (on the syringe and vial) are covered by the hands. And the hands are moving fast. They are doing the job. They aren’t posing for the camera.

Shyam Gollakota, Study Co-Author and Professor, Paul G. Allen School of Computer Science & Engineering, University of Washington

Additionally, the computational model had to be trained to disregard background vials and syringes, focusing solely on the medications in the foreground of the image.

Gollakota added, “AI is doing all that: detecting the specific syringe that the healthcare provider is picking up, and not detecting a syringe that is lying on the table.”

This study highlights the potential of AI and deep learning to enhance safety and efficiency in various healthcare settings, with researchers only beginning to explore its full possibilities, Michaelsen noted.

The study also involved researchers from Carnegie Mellon University and Uganda’s Makerere University, with the Toyota Research Institute developing and testing the system. The research was supported by the Washington Research Foundation, the Foundation for Anesthesia Education and Research, and a National Institutes of Health grant (K08GM153069).

Journal Reference:

Chan, J. et. al. (2024) Detecting clinical medication errors with AI enabled wearable cameras. npj Digital Medicine. doi.org/10.1038/s41746-024-01295-2