Aug 10 2020
An innovative platform based on artificial intelligence (AI) has been developed by experts from the Center for Diagnostics and Telemedicine. This platform has been designed for self-testing services, including medical tasks, such as the study of diagnostic images.
The first working prototype of the platform is hosted on the popular GitHub service, and developers from all over the world can take part in its improvement by adding verification criteria depending on the purpose of the services. Image Credit: Center for Diagnostics and Telemedicine.
The preliminary working model of the AI-based platform is hosted on the famous GitHub service, and developers from around the world can participate in its enhancement by introducing verification standards based on the purpose of the services.
Sergey Morozov, Chief Executive Officer from the Center for Diagnostics and Telemedicine, has discussed this topic at the thematic week devoted to AI. The thematic week was part of the European Congress of Radiology (ECR 2020) program.
Prior to applying an AI-based service to regular clinical practice, it should be tested for technical readiness, and verification should also be made whether it fulfills the stated criteria. It is known as the analytical validation of the algorithm. Only those services that have passed this analytical validation are permitted to be incorporated into medical systems, such as city healthcare.
However, integration is known to be a costly and complex procedure. Therefore, it becomes a hindrance for several groups that cannot ensure the needed speed and precision of the algorithm processing data of the system into which they are incorporated.
Analytical validation is presently being carried out manually. But manual validation can lead to deliberate or accidental deviations from the sanctioned test program. It can also lead to the exploitation of datasets and can possibly put different participants of the test in uneven conditions.
To overcome these issues, automate the verification procedure, and thus ensure the trust of users, experts from the Center for Diagnostic and Telemedicine have designed a new platform that enables AI-based service developers to separately perform initial tests—that is, analytical validation—of their algorithms.
A version of the platform has been hosted on the popular GitHub, and the original version of the service meant for exchanging datasets as well as data analysis outcomes has already been uploaded.
The platform provides an opportunity for the unlimited number of accesses to single samples of data instances from the test set in order to fine-tune algorithms. It has uniform rules of use, and it is possible to test several services simultaneously. At the same time, the platform records the time that the software spends on data processing (time-study), and the developers receive an automatic report on the results of testing.
Sergey Morozov, CEO, Center for Diagnostic and Telemedicine
Automation of the whole procedure on the self-testing platform minimizes the human factor, making it impossible to manipulate the data (to enhance results).
Moreover, the comparison between the verification results of the service and the reference data is fully transparent—the developer can view the kind of metrics that were employed, and can also see how the end result mirrored in the report was computed.
Anyone can take part in improving the platform and add necessary metrics to it, which will be used to evaluate the algorithm's performance for certain medical purposes (for example, for analyzing radiographs or mammograms). However, the addition of the platform will be monitored—the only metrics that have scientific justification will be included in the platform operating on the basis of the Center.
Nikolai Pavlov, Head of Dataset Labeling Conveyor of the Medical Informatics, Radiomics and Radiogenomics Sector, Center for Diagnostics and Telemedicine
Pavlov developed the platform.
The developers of the new platform are inviting AI algorithms developers, researchers, and programmers to participate in updating and enhancing the platform to create a universal, uniform, easy-to-use tool for self-testing of AI algorithms meant for medical purposes in the global community. At present, no such tool is available that is specifically targeted at the clinical implementation of services built on AI technologies.