Source: Zhongshan Ophthalmic Center
Edited by: Tan Rongyu, Wang Dongmei
As science and technology benefit China’s health, the booming of artificial intelligence (AI) in medicine is hopeful to handle the huge problem of medical resource shortage and help achieving state-level strategy of comprehensive health development of the “Healthy China 2030” blueprint. With the support of programs such as the National Key R&D Program of China, a team led by Professor Haotian Lin from the Zhongshan Ophthalmic Center (ZOC) of Sun Yat-sen University has completed the world's first multi-center real world research on the evaluation and supervision strategies of AI medical devices before clinical application, accompanying with 18 domestic and foreign medical establishments, scientific research institutions and enterprises, including Airdoc, a leading enterprise in the field of AI in medicine, and Guangdong Medical Devices Quality Surveillance and Test Institute. On 27th July, 2021, the team has published the article “Application of Comprehensive Artificial Intelligence Retinal Expert (CARE) system: a national real-world evidence study” in
The Lancet Digital Health (impact factor: 24.519), the world’s leading medical journal. The team has also possessed a series of technical patents and obtained the first ophthalmic AI software class III medical device product registration certificate issued by the National Medical Products Administration.
This national real-world study “marking the right development direction”
In recent years, Medical AI has entered the clinical stage. However, due to the complicated environment in the real world, the high-performance deep-learning system are often not acclimatized in the authentic clinical environment, its robustness not ideal and images unidentified, which hinders the clinical transition of Medical AI. By using more than 260,000 color fundus images from medical institutions with different disease characteristics, such as tertiary hospitals, community hospitals and health service institutions, the project team trained CARE, a comprehensive intelligent diagnosis expert of fundus diseases that can identify 14 common fundus abnormalities. CARE is a multi-label deep-learning network, in which the labels and features of various diseases are trained in the same neural network. The model can recognize a variety of fundus abnormalities and correlate the relationship between the characteristics of various diseases. It not only reduces the dependence of model operation on computing resources, but also improves the overall accuracy of diagnosis from 0.921 to 0.952. The CARE model was verified in the authentic clinical environment in 35 medical institutions at different levels in China, completing the world's first multi-center clinical real-world study on medical AI. This work was rated as "marking the right development direction of medical artificial intelligence research" by Amitha Domalpally, director of the world authoritative imaging diagnosis center of the University of Wisconsin, one of the top ten research universities in the United States.
Fig 1. Multi-label deep learning model--CARE
To break through the bottleneck of enhancing the performance and stability of the intelligent diagnosis model, it is needed to use abundant representative data to train the model and conduct multi-center clinical real-world research in a large scale, under the cooperation among medical institutions, corporations and government regulators. In this project uniting ZOC, Airdoc, Medical Devices Quality Surveillance and Test Institute among others, over 260,000 color fundus images covering different disease characteristics were included. Among them, the data for model training are from 16 representative medical institutions of different levels, 35 prospective clinical real-world verification institutions distributed in 28 provinces and cities, including 8 tertiary hospitals, 6 community hospitals and 21 health service agencies across the country. The multi-center clinical real-world research not only improved the performance of the CARE model, but also created the full-link cooperation mode from clinical problems, R&D of intelligent screening and diagnosis model, to the medicine, research, production and management of clinical application.
Fig 2. Distribution of the medical institutes in the real-world research
Real-world research is significant to the evaluation and surveillance of AI medical equipment
How to construct an effective quality evaluation system for AI medical equipment is a crucial link to pushing forward the healthy development of the industry. In addition to the prospective real-time verification in the authentic clinical environment, the team also designed a series of clinical tests, including comparing the performance of the CARE system with 16 ophthalmologists with different years of experience from different regions, and testing the system with color fundus photos from non-Chinese race obtained by camera types not involved in model training. In addition, the team tested CARE using an electronically scanned fundus images taken with a film camera for the first time. The results demonstrated that the CARE system exhibited robust disease recognition ability in clinical real-world validation. Promisingly, the diabetic retinopathy identification module therein has received the first ophthalmic artificial intelligence software class III medical device product registration issued by the National Medical Products Administration (Diabetic Retinopathy Analysis Software, registration certificate for medical device 20203210686), providing a feasible reference scheme for the evaluation and regulation of AI medical devices in China.
Fig 3. CARE exhibited a comparable performance and a better stability than ophthalmologists
With the support of National Key R&D Scheme (2018YFC0116500) and Guangdong Provincial Key R&D Scheme (2018B0109008), the `project was completed in three years with the joint efforts of Zhongshan Ophthalmic Center of Sun Yat-sen University and Beijing Airdoc Technology Co., Ltd. (Airdoc) and 18 medical institutions, enterprises and research institutions at home and abroad. Ikang Guobin Healthcare Group, Baodao Glasses and Ping An Insurance also provided assistance during the implementation of the project. The main persons completing the project include Professor Haotian Lin of the ZOC, Sun Yat-sen University (final corresponding author, PI), Dr. and postdoctoral Duoru Lin (first author); and Professor Yuzhong Chen (joint corresponding author), Dr. Jianhao Xiong (joint first author) and engineer Congxin Liu (joint first author) of Airdoc.
Link to the paper:
https://www.thelancet.com/journals/landig/article/PIIS2589-7500(21)00086-8/fulltext#figures
Link to the comment:
https://www.thelancet.com/journals/landig/article/PIIS2589-7500(21)00140-0/fulltext
