Automated machine learning (AutoML) models for diabetic retinopathy (DR) image classification from handheld retinal images

Purpose
To create and validate automated deep learning models for DR that are trained on handheld retinal images for community-based DR screening program (DRSP) in the Philippines.

Methods
AutoML Vision (Google Cloud) models were generated based on previously acquired 2-field retinal images (macula and disc centered, 1,600 images) from the Philippine DRSP. Image labeling was based on the International DR and DME classification obtained from primary grades and secondary adjudications by a reading center (RC). Images for the initial model were split 8-1-1 for training, validation and testing to detect referable DR [(refDR), defined as moderate nonproliferative DR or worse or any level of diabetic macular edema (DME). External testing of the autoML model was performed using a published image set (N=225 eyes) using the same devices in the same population, evaluated by the same RC. Sensitivity and specificity (SN/SP) for refDR were calculated.

Results
Training set distribution of DR severity by RC: no DR 66.0%, mild NPDR 10.7%, moderate NPDR 7.9%, severe NPDR 3.3%, PDR 5.6%, ungradable 6.5%. DME severity was: no DME 83.6%, DME 6.3%,center involved DME 7.4%, ungradable 2.7%. RefDR was present in 18.5% of images. Area under the precision-recall curve (AUPRC) was 0.947 (figure 1). The model’s overall accuracy for RefDR was 89.4%. External testing set DR/DME distribution: no DR 54.2%, mild NPDR 17.8%, moderate NPDR 9.8%, severe NPDR 3.3%, PDR 5.8%, ungradable 1.8%. DME severity was: no DME 62.7%, DME 6.2%, center involved DME 19.1%, ungradable 12.0%. RefDR was present in 39.1% of images. SN/SP for refDR on the external test set was 0.94/0.81. Table 1 shows a comparison with reported metrics from FDA approved algorithms.

Conclusions
This study demonstrates the accuracy and feasibility of autoML models for the identification of refDR developed for a DRSP using handheld retinal imaging in a low-resource setting community program. The performance approaches published diagnostic accuracy metrics of commercial models used for DRSP. Potentially, the use of autoML may increase access to machine learning models that may be adapted for specific programs that are guided by clinicians to rapidly address disparities in patient care.