AI Tool Enhances Detection of Rare Diseases Using Common Data

AI shows considerable potential for medical diagnosis through imaging data, though it often struggles with rare diseases due to limited training data. Professor Klaus-Robert Müller’s team at TU Berlin/BIFOLD, in collaboration with Charité —Universitätsmedizin Berlin, has introduced a new approach to address this challenge by training an AI model on common findings to improve the detection of rare diseases. The study was published in the New England Journal of Medicine AI (NEJM AI).

AI is widely used across various medical fields and shows immense potential in assisting doctors with disease diagnosis through imaging data. However, training AI models requires large datasets, which are typically only available in adequate amounts for common diseases.

It’s as if a family doctor only had to diagnose coughs, runny noses, and sore throats. The actual challenge is to also detect the less common diseases, which current AI models often overlook or misclassify.

Frederick Klauschen, Professor and Director, Institute of Pathology, Ludwig-Maximilians-Universität München

The new model can reliably detect rare diseases using training data from common findings alone. This advancement has the potential to significantly improve diagnostic accuracy and reduce pathologists' workloads in the future.

Learning From Normality

The new approach utilizes anomaly detection: by accurately characterizing normal tissue and common diseases, the model learns to identify and highlight deviations without needing specific training on rare cases.

For the study, researchers compiled two large datasets of microscopic images from gastrointestinal biopsy sections, along with their diagnoses. Approximately 90 % of these datasets consisted of the ten most common findings, including normal tissues and conditions like chronic gastritis, while the remaining 10 % encompassed 56 disease types, many of which are cancers. In total, the model was trained and evaluated on 17 million histological images from 5,423 cases.

We compared various technical approaches, and our best model detected, with a high degree of reliability, a broad range of rarer pathologies of the stomach and colon, including rare primary or metastasizing cancers. To our knowledge, no other published AI tool is capable of doing this.

Klaus-Robert Müller, Technical University of Berlin

Additionally, the AI uses heatmaps to mark the locations of anomalies within the tissue section in color.

Significantly Easing the Diagnosis Workload

By distinguishing normal findings and common diseases and identifying anomalies, the new AI model—which will continue to be refined over time—has the potential to offer valuable support to doctors. While pathologists still need to confirm the identified diseases, “doctors can save a lot of time because normal findings and a certain proportion of the diseases can be automatically diagnosed by AI. This applies to around a quarter to a third of cases,” explains Klauschen. “And in the remaining cases, AI can facilitate case prioritization and reduce missed diagnoses. This would represent huge progress.”

Journal Reference:

Dippel, J., et al. (2024) AI-Based Anomaly Detection for Clinical-Grade Histopathological Diagnostics. NEJM AI. doi.org/10.1056/aioa2400468.