Comparative evaluation of hybrid deep learning models for mmWave radar-based multihand gesture recognition

According to data from the World Federation of the Hearing Impaired and the World Health Organization, in the world, there are about 72 million people who suffer from hearing impairment, and the total population of hearing-impaired people is 360 million, out of which 32 million are children. This group of individuals uses hand gestures as the main form of communication. The objective of hand gesture recognition is the acquisition of the hand-gesture data using a series of sensors. Many vision-based methods have been developed for HGR with an impressive level of accuracy; however, its performance is often impaired by the variation in lighting, occlusion and privacy issues. These shortcomings correspond to huge practical impediments for real-time applications. The millimetre-wave (mmWave) radar sensing dataset, which was deployed in this work using the AWR1642BOOST platform from Texas Instruments, is a potential alternative, given its tolerance to lighting conditions, its ability to see through some materials and its privacy-enhancing properties. The aim of this work is to realise high-performance classification of multihand gestures using mmWave radar point cloud data. This paper presents the comparative analysis of hybrid deep learning models for mmWave radar-based multihand gesture recognition based on a publicly available dataset. In this study, we trained, tested and validated a 1D-CNN with a gated MLP, a pure 1D-CNN and a CNN extractor with bidirectional LSTM. The CNN–LSTM model has the least number of parameters, 1,020,941, while achieving the best overall performance. It achieved a training accuracy of 99.73 a validation accuracy of 97.027.73 recall and F1-score were 0.9759, 0.9791 and 0.9773, respectively. Cross-validation results also indicate its robustness, such that the mean accuracy is 93.9 and the standard deviation is 0.0146.