Real Time Automated Speech Recognition Transcription and Sign Language Character Animation on Learning Media

Komang Kurniawan Widiartha; Ketut Agustini; I Made Tegeh; I Wayan Sukra Warpala

doi:10.23887/janapati.v13i3.85065

Authors

Komang Kurniawan Widiartha STMIK STIKOM INDONESIA https://orcid.org/0000-0001-8761-1784
Ketut Agustini Universitas Pendidikan Ganesha
I Made Tegeh Universitas Pendidikan Ganesha
I Wayan Sukra Warpala Universitas Pendidikan Ganesha

DOI:

https://doi.org/10.23887/janapati.v13i3.85065

Keywords:

Learning Media, ASR Transcription, Sign Language Animation, Deaf Education, Educational Technology, Educational Inclusion

Abstract

Inclusive education for deaf students requires a technology approach to address communication and comprehension challenges. This study aims to develop innovative learning media that integrates real-time ASR (Automated Speech Recognition) transcription technology and sign language character animation to improve accessibility and comprehension of materials for deaf students. This learning media receives input from live voice, voice from learning videos, and text inputted by teachers. Using the Google Cloud API-based ASR transcription module, voice and text are converted into written text, broken down into vocabulary for sign language animation search. The search is carried out using an interpolation algorithm in the sign language animation asset database, allowing the display of animations relevant to the transcribed vocabulary.

The development process follows the ADDIE instructional design model, starting with needs analysis and ending with implementation and evaluation. The analysis stage includes data collection through teacher interviews, classroom observations, and curriculum reviews. The media design is designed to meet the specific needs of deaf students, while development and implementation focus on technology integration and effective material delivery. Evaluation is carried out to assess the effectiveness of the media in improving student understanding and participation. The study's results showed that this learning media can improve deaf students' understanding of the material and increase their involvement in the learning process. ASR technology and sign language animation contribute significantly to making learning materials more accessible and understandable, supporting the goals of inclusive education.

References

H. Mansur, A. H. Utama, M. H. M. Yasin, N. P. Sari, K. A. Jamaludin, and F. Pinandhita, “Development of Inclusive Education Learning Design in the Era of Society 5.0,” Soc. Sci., p. 496, 2023, doi: 10.4337/9781788974912.S.41.

S. Agung, A. N. Khoirunisa, and S. Suryaningsih, “Tantangan Guru Sekolah Luar Biasa Pada Pembelajaran Ipa Di Abad 21,” Alotrop, vol. 6, no. 1, pp. 43–52, 2022, doi: 10.33369/alo.v6i1.21089.

I. Azizah, “Metode Pengajaran Anak Berkebutuhan Khusus Di Sekolah Luar Biasa (Slb),” Pena Kreat. J. Pendidik., vol. 11, no. 1, pp. 42–47, 2022, doi: 10.29406/jpk.v11i1.1953.

D. A. Dewantoro, R. F. Pradipta, M. Efendi, A. Huda, and M. H. M. Yasin, “Availability and Utilization Analysis Learning Media at Inclusive Schools in Malang City,” Proc. Int. Conf. Inf. Technol. Educ. (ICITE 2021), vol. 609, no. Icite, pp. 12–21, 2023, doi: 10.2991/assehr.k.211210.003.

V. Yunus, A. Zakso, A. T. Priyadi, and A. Hartoyo, “Pendidikan Inklusif Pada Kurikulum Merdeka,” J. Pendidik. DASAR PERKHASA J. Penelit. Pendidik. Dasar, vol. 9, no. 2, pp. 313–327, 2023, doi: 10.31932/jpdp.v9i2.2270.

Kementrian Sekretariat Negara RI, “Undang-Undang Republik Indonesia Nomor 8 Tahun 2016 Tentang Penyandang Disabilitas,” no. 243, pp. 1–5, 2016.

N. Made et al., Problems of Teachers in Inclusive Class Learning in Public Elementary School Bengkala Buleleng, vol. 1. Atlantis Press SARL, 2023. doi: 10.2991/978-2-494069-29-9.

D. Tresnawati, R. Setiawan, N. Alawiah, D. Heryanto, and S. Rahayu, “The development of sign language learning media using multimedia systems,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1098, no. 3, p. 032059, 2021, doi: 10.1088/1757-899x/1098/3/032059.

N. Putu, A. Pirdayanti, N. K. Anggriani, K. Jepri, and K. Jaya, “ Kolok-Indonesian-English ” Through Smart Kolok Application as a Media for Learning and Preserving a Natural Sign Language in Bengkala Village. Atlantis Press SARL, 2022. doi: 10.2991/978-2-494069-29-9.

O. E. Ghoul, “Virtual reality for educating Sign Language using signing avatar: The future of creative learning for deaf students,” IEEE Global Engineering Education Conference, EDUCON, vol. 2022. pp. 1269–1274, 2022. doi: 10.1109/EDUCON52537.2022.9766692.

D. P. Saridewi, I. W. Santyasa, I. M. Sutajaya, and N. N. Parwati, “application of the mesatua method using cartoon media in Hindu religious learning to develop student independence with disabilities,” Int. J. Health Sci. (Qassim)., no. 3, pp. 4356–4366, 2022, doi: 10.53730/ijhs.v6ns5.10795.

L. S. Wilyanti, L. Larlen, and S. Wulandari, “Transkripsi Sastra Lisan Melayu Jambi sebagai Alternatif Bahan Ajar Sastra di Perguruan Tinggi,” J. Ilm. Univ. Batanghari Jambi, vol. 22, no. 1, p. 247, 2022, doi: 10.33087/jiubj.v22i1.1889.

J. Zhang et al., “Intelligent speech technologies for transcription, disease diagnosis, and medical equipment interactive control in smart hospitals: A review,” Comput. Biol. Med., vol. 153, no. January, p. 106517, 2023, doi: 10.1016/j.compbiomed.2022.106517.

M. P. Aji and U. G. Aeman, “Aplikasi Speech Recognition untuk Audio Forensik,” J. Media Pratama, vol. 17, no. 1, pp. 61–73, 2023.

M. Carrier, “Automated speech recognition in language learning: Potential models, benefits and impact,” Training, Lang. Cult., vol. 1, no. 1, pp. 46–61, 2017, doi: 10.29366/2017tlc.1.1.3.

N. Sutomo, “The Implementation of Automated Speech Recognition (ASR) in ELT Classroom: A Systematic Literature Review from 2012-2023,” VELES (Voices English Lang. Educ. Soc., vol. 7, no. 3, pp. 816–828, 2024, doi: 10.29408/veles.v7i3.23978.

R. Arakawa, H. Yakura, and M. Goto, “BeParrot: Efficient Interface for Transcribing Unclear Speech via Respeaking,” Int. Conf. Intell. User Interfaces, Proc. IUI, pp. 832–840, 2022, doi: 10.1145/3490099.3511164.

X. Li, S. Yang, and H. Guo, “Application of virtual human sign language translation based on speech recognition,” Speech Commun., vol. 152, no. March, p. 102951, 2023, doi: 10.1016/j.specom.2023.06.001.

T. F. Doni, D. M. Midyanti, and S. Bahri, “Implementasi Metode Interpolation Search Pada Kamus Bahasa Indonesia Ke Bahasa Dayak Kalimantan Barat Berbasis Android,” J. Komput. dan Apl., vol. 09, no. 03, pp. 436–445, 2021.

R. M. Branch, Approach, Instructional Design: The ADDIE, vol. 53, no. 9. 2009.