Multimodal deep learning model based on ECG and clinical notes for arrhythmia classification

Electrocardiograms (ECGs) are non-invasive tools used to monitor the heart’s electrical activity. It captures the heart’s depolarization and repolarization cycles by measuring the tiny voltage fluctuations on the skin. Analyzing this rhythm requires intensive effort from clinicians. This paper introduce a novel end-to-end multi-modal deep learning model to process 12-lead ECG signals with patient clinical notes to assist cardiologists in classifying eight types of arrhythmia. The ECG signal is processed using a convolutional backbone composed of residual 1D convolutional blocks, a convolutional block attention module, and a bi-directional GRU to capture long-range temporal dependencies. In parallel, 13 clinical features extracted from clinical notes are embedded using an auxiliary multilayer perceptron network. The two modalities are combined through late fusion, followed by a fully connected layer for arrhythmia classification. The model is trained and evaluated on a public dataset. It achieves an accuracy of 92.8% and a macro F1-score of 84.6%, representing +4% increase in accuracy and +9% improvement in macro F1-score compared to processing ECG signals alone. Furthermore, it also outperforms a baseline random forest by a substantial margin. Error analysis shows that the clinical notes help to improve the discriminatory ability of the model for minority and morphologically similar rhythms, particularly Sinus Arrhythmia, Sinus Bradycardia, and Sinus Rhythm. The proposed model demonstrates the clinical potential of fusing waveform morphology with contextual patient data can enable accurate diagnostic tools.