A cross-modal feature fusion method to diagnose macular fibrosis in neovascular age-related macular degeneration

This paper presents a novel Cross-modal Fusion Method (CrossFM) for automated classification of macular fibrosis in neovascular Age-related Macular Degeneration (nAMD). Firstly, a CoTransNet network is proposed to combine convolution and attention blocks for exploring the low-frequency global and high-frequency local information of lesion region for each modality. Then a dual-branch CoTransNet network is adopted to accept input data from randomly paired Optical Coherence Tomography (OCT) and Ultra-Wide-angle Fundus (UWF) images of the same eye in each single branch. Secondly, we propose a cross-modal attention strategy to integrate characteristics across various modalities to optimize lesion descriptive abilities with dynamically adaptive weights for each branch. Finally, image-level results are aggregated to obtain eye-level diagnosis results using high-probability reliability maps. Experimental results verify the effectiveness of the proposed method and even to the level of senior ophthalmologists. CrossFM also outperforms some state-of-the-art multimodal methods on both constructed and public datasets.