Abstract

This research develops a real-time Greek sign language (GSL) recognition system using machine learning and computer vision techniques to assist communication for the deaf community. The hand tracking module from the CVZone library was utilized for hand detection and tracking in video feeds. A dataset of 37,589 images capturing 24 Greek alphabet letter gestures was collected from different users under varying conditions. This dataset was used to train a deep neural network model based on the MobileNetV2 architecture. The model achieved 96.86% accuracy on the validation set during training. On the final test set, it obtained 95.66% accuracy in classifying and recognizing GSL hand gestures. Compared to prior rule-based and fuzzy clustering approaches, the model demonstrated significantly improved recognition performance. The high accuracy indicates that combining robust computer vision techniques for hand tracking with deep convolutional neural networks can be an effective approach for real-time GSL recognition. This system has the potential to facilitate communication and accessibility for the deaf and hard-of-hearing community. Further work should focus on expanding the vocabulary beyond individual letters to incorporate common words and phrases, as well as optimizing the system's real-time performance by reducing processing lags. Overall, this research demonstrates a promising machine learning and computer vision-based system for automatic GSL recognition that can aid the deaf community.

Keyword

SLR, ASLR, RNN and CNN.

Cite this article

Abid MA, Pillai N, Ahammed N.Greek sign language recognition using machine learning. International Journal of Advanced Computer Research. 2024;14(66):1-17. DOI:10.19101/IJACR.2023.1362022

Refference

[1]Obi Y, Claudio KS, Budiman VM, Achmad S, Kurniawan A. Sign language recognition system for communicating to people with disabilities. Procedia Computer Science. 2023; 216:13-20.

[2]Riad AM, Elminir HK, Shohieb SM. Hand gesture recognition system based on a geometric model and rule based classifier. British Journal of Applied Science & Technology. 2014; 4(9):1432-44.

[3]Mariappan HM, Gomathi V. Real-time recognition of Indian sign language. In international conference on computational intelligence in data science (ICCIDS) 2019 (pp. 1-6). IEEE.

[4]Wu J, Sun L, Jafari R. A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors. IEEE Journal of Biomedical and Health Informatics. 2016; 20(5):1281-90.

[5]Rekha J, Bhattacharya J, Majumder S. Hand gesture recognition for sign language: a new hybrid approach. In proceedings of the international conference on image processing, computer vision, and pattern recognition (IPCV) 2011 (pp. 1-7). WorldComp.

[6]Huu PN, Phung NT. Hand gesture recognition algorithm using SVM and HOG model for control of robotic system. Journal of Robotics. 2021; 2021:1-3.

[7]Shinde P, Shinde P, Shinde S, Shinde S, Shinde S. Augmented reptile feeder. In Pune section international conference (PuneCon) 2022 (pp. 1-4). IEEE.

[8]Ismail MH, Dawwd SA, Ali FH. A review on Arabic sign language recognition. Journal of Advances in Computer and Electronics Engineering. 2021; 6(12):1-12.

[9]Madhubala B, Sarathambekai S, Vairam T, Seelan KS, Sathya RS, Swathy AR. Pre-trained convolutional neural network model based pneumonia classification from chest X-Ray images. In proceedings of the international conference on smart data intelligence (ICSMDI 2021) 2021(pp.1-8).

[10]Michele A, Colin V, Santika DD. Mobilenet convolutional neural networks and support vector machines for palmprint recognition. Procedia Computer Science. 2019; 157:110-7.

[11]Carney M, Webster B, Alvarado I, Phillips K, Howell N, Griffith J, et al. Teachable machine: approachable web-based tool for exploring machine learning classification. In extended abstracts of the 2020 CHI conference on human factors in computing systems 2020 (pp. 1-8). ACM.

[12]Dogo EM, Afolabi OJ, Nwulu NI, Twala B, Aigbavboa CO. A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. In international conference on computational techniques, electronics and mechanical systems (CTEMS) 2018 (pp. 92-9). IEEE.

[13]Patro VM, Patra MR. Augmenting weighted average with confusion matrix to enhance classification accuracy. Transactions on Machine Learning and Artificial Intelligence. 2014; 2(4):77-91.

[14]Agarwal A, Sharma P, Alshehri M, Mohamed AA, Alfarraj O. Classification model for accuracy and intrusion detection using machine learning approach. Peer J Computer Science. 2021; 7:e437.

[15]Wadhawan A, Kumar P. Deep learning-based sign language recognition system for static signs. Neural Computing and Applications. 2020; 32:7957-68.

[16]Wen F, Zhang Z, He T, Lee C. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nature Communications. 2021; 12(1):1-13.

[17]Rastgoo R, Kiani K, Escalera S. Sign language recognition: a deep survey. Expert Systems with Applications. 2021; 164:113794.

[18]Rastgoo R, Kiani K, Escalera S. Hand sign language recognition using multi-view hand skeleton. Expert Systems with Applications. 2020; 150:113336.

[19]Wadhawan A, Kumar P. Sign language recognition systems: a decade systematic literature review. Archives of Computational Methods in Engineering. 2021; 28:785-813.

[20]Al-hammadi M, Muhammad G, Abdul W, Alsulaiman M, Bencherif MA, Alrayes TS, et al. Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access. 2020; 8:192527-42.

[21]Corballis MC. From mouth to hand: gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences. 2003; 26(2):199-208.

[22]Oyedotun OK, Khashman A. Deep learning in vision-based static hand gesture recognition. Neural Computing and Applications. 2017; 28(12):3941-51.

[23]Nagi J, Ducatelle F, Di CGA, Cireşan D, Meier U, Giusti A, et al. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In international conference on signal and image processing applications (ICSIPA) 2011 (pp. 342-7). IEEE.

[24]Rioux-maldague L, Giguere P. Sign language fingerspelling classification from depth and color images using a deep belief network. In Canadian conference on computer and robot vision 2014 (pp. 92-7). IEEE.

[25]Huang J, Zhou W, Li H, Li W. Sign language recognition using 3d convolutional neural networks. In international conference on multimedia and expo (ICME) 2015 (pp. 1-6). IEEE.

[26]Huang J, Zhou W, Li H, Li W. Sign language recognition using real-sense. In China summit and international conference on signal and information processing (ChinaSIP) 2015 (pp. 166-70). IEEE.

[27]Pigou L, Dieleman S, Kindermans PJ, Schrauwen B. Sign language recognition using convolutional neural networks. In computer vision-ECCV workshops: Zurich, Switzerland, Proceedings, Part I 13 2015 (pp. 572-8). Springer International Publishing.

[28]Molchanov P, Gupta S, Kim K, Pulli K. Multi-sensor system for drivers hand-gesture recognition. In 11th IEEE international conference and workshops on automatic face and gesture recognition (FG) 2015 (pp. 1-8). IEEE.

[29]Tang A, Lu K, Wang Y, Huang J, Li H. A real-time hand posture recognition system using deep neural networks. ACM Transactions on Intelligent Systems and Technology (TIST). 2015; 6(2):1-23.

[30]Yang S, Zhu Q. Video-based Chinese sign language recognition using convolutional neural network. In 9th international conference on communication software and networks (ICCSN) 2017 (pp. 929-34). IEEE.

[31]Tushar AK, Ashiquzzaman A, Islam MR. Faster convergence and reduction of overfitting in numerical hand sign recognition using DCNN. In region 10 humanitarian technology conference (R10-HTC) 2017 (pp. 638-41). IEEE.

[32]Bheda V, Radpour D. Using deep convolutional networks for gesture recognition in american sign language. arXiv preprint arXiv:1710.06836. 2017.

[33]Rao GA, Syamala K, Kishore PV, Sastry AS. Deep convolutional neural networks for sign language recognition. In conference on signal processing and communication engineering systems (SPACES) 2018 (pp. 194-7). IEEE.

[34]Koller O, Zargaran S, Ney H, Bowden R. Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. International Journal of Computer Vision. 2018; 126:1311-25.

[35]Kumar EK, Kishore PV, Kumar MT, Kumar DA. 3D sign language recognition with joint distance and angular coded color topographical descriptor on a 2–stream CNN. Neurocomputing. 2020; 372:40-54.

[36]Prabhu R. Understanding of convolutional neural network (CNN)-deep learning. Medium Com. 2018:1-11.

[37]Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M. Real-time computer vision-based Bengali sign language recognition. In 17th international conference on computer and information technology (ICCIT) 2014 (pp. 192-7). IEEE.

[38]Uddin MA, Chowdhury SA. Hand sign language recognition for Bangla alphabet using support vector machine. In international conference on innovations in science, engineering and technology (ICISET) 2016 (pp. 1-4). IEEE.

[39]Rao GA, Kishore PV. Selfie video based continuous Indian sign language recognition system. Ain Shams Engineering Journal. 2018; 9(4):1929-39.

[40]Zhou H, Zhou W, Zhou Y, Li H. Spatial-temporal multi-cue network for continuous sign language recognition. In proceedings of the AAAI conference on artificial intelligence 2020 (pp. 13009-16).

[41]Sarhan N, Wilms C, Closius V, Brefeld U, Frintrop S. Hands in focus: sign language recognition via top-down attention. In international conference on image processing (ICIP) 2023 (pp. 2555-9). IEEE.

[42]Marais M, Brown D, Connan J, Boby A. Spatiotemporal convolutions and video vision transformers for signer-independent sign language recognition. In international conference on artificial intelligence, big data, computing and data communication systems (icABCD) 2023 (pp. 1-6). IEEE.

[43]Tarrés L, Gállego GI, Duarte A, Torres J, Giró-i-nieto X. Sign language translation from instructional videos. In proceedings of the conference on computer vision and pattern recognition 2023 (pp. 5624-34). IEEE.

[44]Laines D, Gonzalez-mendoza M, Ochoa-ruiz G, Bejarano G. Isolated sign language recognition based on tree structure skeleton images. In proceedings of the conference on computer vision and pattern recognition 2023 (pp. 276-84). IEEE.

[45]Arkushin RS, Moryossef A, Fried O. Ham2pose: Animating sign language notation into pose sequences. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2023 (pp. 21046-56).

[46]Papadimitriou K, Sapountzaki G, Vasilaki K, Efthimiou E, Fotinea SE, Potamianos G. SL-REDU GSL: a large greek sign language recognition corpus. In international conference on acoustics, speech, and signal processing workshops (ICASSPW) 2023 (pp. 1-5). IEEE.

[47]Papadimitriou K, Potamianos G. Sign language recognition via deformable 3d convolutions and modulated graph convolutional networks. In international conference on acoustics, speech and signal processing (ICASSP) 2023 (pp. 1-5). IEEE.

[48]Naz N, Sajid H, Ali S, Hasan O, Ehsan MK. Signgraph: an efficient and accurate pose-based graph convolution approach toward sign language recognition. IEEE Access. 2023; 11:19135-47.

[49]Al-qaisy NE, Al-kaseem BR, Al-dunainawi Y. AI-based portable gesture recognition system for hearing impaired people using wearable sensors. In 15th international conference on developments in eSystems engineering (DeSE) 2023 (pp. 33-8). IEEE.

[50]Mahyoub M, Natalia F, Sudirman S, Mustafina J. Sign language recognition using deep learning. In 15th international conference on developments in eSystems engineering (DeSE) 2023 (pp. 184-9). IEEE.