2MLMD: multi-modal leap motion dataset for home automation hand gesture recognition systems

In the field of multi-modal hand gesture recognition (HGR) systems, the scarcity of comprehensive datasets poses a significant limitation to the development and evaluation of models. Existing datasets often lack diversity in terms of gestures, environmental conditions, and demographic representations, limiting the generalizability of algorithms. Recognizing the crucial need to fill this gap, our research aims to contribute a novel and extensive multi-modal HGR dataset named multi-modal leap motion dataset (2MLMD). The proposed dataset simplifies home control through 30 commands (24 dynamic and 6 static) executed using one or two hands through both modalities (skeletal and depth) delivered by a leap motion controller. It covers challenges involving participants of various genders and ages, addressing different scenes and backgrounds. To highlight the effectiveness of our challenging dataset, we analyze the hand gesture classification results adapted to each modality using both traditional and deep models. Extensive experiments show that for both skeletal and depth data, deep learning models outperform traditional handcrafted approaches. Specifically, the bidirectional long short-term memory deep learning model achieves 88% of accuracy on skeletal data, compared to 84% for statistical features with a support vector machine. For depth data, the InceptionV3 deep learning model reaches 93% of accuracy.