Sign Language Recognition Based on Geometric Features Using Deep Learning
DOI:
https://doi.org/10.23887/janapati.v13i2.82103Keywords:
Sign Language Recognition, LSTM Model, Geometric FeaturesAbstract
Sign language plays a crucial role in facilitating communication among individuals with hearing impairments. In Indonesia, the deaf community often rely on BISINDO (Indonesian Sign Language) to communicate amongst themselves. People who are unfamiliar with sign language will face difficulties. This research aims to develop a system for recognizing sign language using geometric features extracted from hand joint coordinates using Google's MediaPipe Hands framework. The dataset contains 12 common words, each recorded 30 times with 30 frames recorded for each instance. This will facilitate communication between deaf and hearing individuals. We conducted tests on LSTM- Geometric and CNN1D- Geometric models using geometric features, and on CNN-LSTM-Spatial and CNN1D-LSTM-Spatial models using spatial features. The results indicate that the LSTM model with geometric features achieved the highest accuracy of 99%. Geometric features have been shown to be more effective than spatial features for classifying sign language gestures.
References
Sihananto, A. N., Safitri, E. M., Maulana, Y., Fakhruddin, F., & Yudistira, M. E. (2023). Indonesian Sign Language Image Detection Using Convolutional Neural Network (CNN) Method. Inspiration: Jurnal Teknologi Informasi dan Komunikasi, 13(1), 13-21
Bai, Y., & Bruno, D. (2020). Addressing Communication Barriers Among Deaf Populations Who Use American Sign Language in Hearing-Centric Social Work Settings. Polymer Journal, 18, 37-50.
Luft, Pamela. (2000). Communication barriers for deaf employees: Needs assessment and problem-solving strategies. Work (Reading, Mass.). 14. 51-59.
Z. R. Saeed, Z. B. Zainol, B. B. Zaidan and A. H. Alamoodi, "A Systematic Review on Systems-Based Sensory Gloves for Sign Language Pattern Recognition: An Update From 2017 to 2022," in IEEE Access, vol. 10, pp. 123358-123377, 2022
Y. -C. Lai, P. -Y. Huang and T. -S. Horng, "Wi-Fi SIMO Radar for Deep Learning-Based Sign Language Recognition," in IEEE Microwave and Wireless Technology Letters, vol. 34, no. 6, pp. 825-828, June 2024
Chong T-W, Lee B-G. American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach. Sensors. 2018; 18(10):3554.
T. Li, Y. Yan and W. Du, "Sign Language Recognition Based on Computer Vision," 2022 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), Dalian, China, 2022
Westlund, J. K., D'Mello, S. K., & Olney, A. M. (2015). Motion Tracker: Camera-Based Monitoring of Bodily Movements Using Motion Silhouettes. PLOS ONE, 10(6), e0130293
Aloysius, N., & Geetha, M. (2020). Understanding vision-based continuous sign language recognition. Multimedia Tools and Applications, 79(33), 22177-22209
D. Fan, H. Lu, S. Xu and S. Cao, "Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition," in IEEE Sensors Journal, vol. 21, no. 23, pp. 27026-27036, 1 Dec.1, 2021
M. A. A. Razak, F. Y. A. Rahman, R. Mohamad, S. Shahbuddin, Y. W. M. Yusof and S. I. Suliman, "Hand Gesture Recognition based on Convolution Neural Network (CNN) and Support Vector Machine (SVM)," 2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC), Shah Alam, Malaysia, 2023, pp. 123-126.
X. -R. Song, J. Yang, S. -M. Zhang and S. Gao, "Research on Gesture Recognition Method Based on Image Processing and SIFT-SVM," 2021 China Automation Congress (CAC), Beijing, China, 2021, pp. 526-531.
P. Pruthvi and J. Geetha, "Convolution Neural Network for Predicting Alphabet Sign Language and Comparative Performance Analysis of CNN, KNN, and SVM Algorithms," 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT), Bangalore, India, 2022, pp. 1-6.
T. Shanableh, "Two-Stage Deep Learning Solution for Continuous Arabic Sign Language Recognition Using Word Count Prediction and Motion Images," in IEEE Access, vol. 11, pp. 126823-126833, 2023, doi: 10.1109/ACCESS.2023.3332250.
J. Shin, A. S. M. Miah, K. Suzuki, K. Hirooka and M. A. M. Hasan, "Dynamic Korean Sign Language Recognition Using Pose Estimation Based and Attention-Based Neural Network," in IEEE Access, vol. 11, pp. 143501-143513, 2023.
O. Koller, N. C. Camgoz, H. Ney and R. Bowden, "Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 9, pp. 2306-2320, 1 Sept. 2020.
A. Agarwal, R. Sreemathy, M. Turuk, J. Jagdale and V. Kumar, "Indian Sign Language Recognition using Skin Segmentation and Vision Transformer," 2023 IEEE 20th India Council International Conference (INDICON), Hyderabad, India, 2023, pp. 857-862, doi: 10.1109/INDICON59947.2023.10440818.
S. Reshna and M. Jayaraju, "Spotting and recognition of hand gesture for Indian sign language recognition system with skin segmentation and SVM," 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, India, 2017, pp. 386-390, doi: 10.1109/WiSPNET.2017.8299784.
U. Rastogi, A. Pandey and V. Kumar, "Skin Segmentation and SVM for Identification and Spotlighting of Hand Gesture for ISLR System," 2023 6th International Conference on Information Systems and Computer Networks (ISCON), Mathura, India, 2023, pp. 1-5.
Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60(2), 91-110.
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M., Lee, J., Chang, W.-T., Hua, W., Georg, M., & Grundmann, M. (2019). MediaPipe: A Framework for Perceiving and Processing Reality. In Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019.
Cao, Z., Hidalgo, G., Simon, T., Wei, S. E., & Sheikh, Y. (2019). OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1), 172-186.
J. Shin, A. S. M. Miah, K. Suzuki, K. Hirooka and M. A. M. Hasan, "Dynamic Korean Sign Language Recognition Using Pose Estimation Based and Attention-Based Neural Network," in IEEE Access, vol. 11, pp. 143501-143513, 2023, doi: 10.1109/ACCESS.2023.3343404.
Ko, S.-K., Kim, C. J., Jung, H., & Cho, C. (2019). Neural Sign Language Translation Based on Human Keypoint Estimation. Applied Sciences, 9(13), 2683
Naidu, G., Zuva, T., Sibanda, E.M. (2023). A Review of Evaluation Metrics in Machine Learning Algorithms. In: Silhavy, R., Silhavy, P. (eds) Artificial Intelligence Application in Networks and Systems. CSOC 2023. Lecture Notes in Networks and Systems, vol 724. Springer, Cham.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Eko Mulyanto Yuniarno
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with Janapati agree to the following terms:- Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. (See The Effect of Open Access)
