Kode Ganda Tunarungu di Live Caption: Interaksi Bahasa Isyarat–Teks Otomatis dan Perbaikan Kesalahan
DOI:
https://doi.org/10.71094/jmsh.v2i2.363Keywords:
live caption, tunarungu, bahasa isyarat, automatic speech recognition, komunikasi multimodalAbstract
Perkembangan teknologi automatic speech recognition (ASR) telah memungkinkan penggunaan live caption sebagai sarana komunikasi bagi komunitas tunarungu dan hard-of-hearing. Meskipun teknologi ini meningkatkan akses terhadap informasi lisan, caption otomatis masih sering mengandung kesalahan transkripsi yang dapat memengaruhi pemahaman pengguna. Penelitian ini bertujuan untuk menganalisis fenomena kode ganda dalam penggunaan live caption oleh pengguna tunarungu, khususnya dalam interaksi antara bahasa isyarat dan teks otomatis serta strategi pengguna dalam menghadapi kesalahan caption. Penelitian ini menggunakan pendekatan kualitatif dengan desain studi interaksi multimodal. Data dikumpulkan melalui observasi interaksi komunikasi yang menggunakan live caption, perekaman video, serta wawancara mendalam dengan partisipan tunarungu. Analisis data dilakukan melalui identifikasi kesalahan caption, analisis interaksi multimodal, dan analisis tematik terhadap strategi pengguna dalam menafsirkan caption. Hasil penelitian menunjukkan bahwa pengguna tunarungu tidak hanya mengandalkan teks caption sebagai sumber informasi utama, tetapi juga memanfaatkan bahasa isyarat dan konteks visual sebagai bagian dari proses pemahaman komunikasi. Interaksi antara bahasa isyarat dan teks caption membentuk praktik kode ganda yang memungkinkan pengguna menafsirkan atau memperbaiki kesalahan caption selama percakapan berlangsung. Temuan ini menunjukkan bahwa pemahaman komunikasi dalam penggunaan live caption bersifat multimodal dan tidak hanya ditentukan oleh akurasi sistem caption. Penelitian ini memberikan kontribusi konseptual terhadap pengembangan teknologi caption yang lebih adaptif terhadap praktik komunikasi komunitas tunarungu.
References
Asrifan, A., Shafa, S., Pratiwi, W. R., et al. (2025). Bridging gaps: AI and NLP approaches for inclusive language learning. Advances in Computational Intelligence and Robotics Book Series. https://doi.org/10.4018/979-8-3693-7260-9.ch006
Bell, J.-M. (2007). Enhancing accessibility through correction of speech recognition errors. ACM SIGACCESS Accessibility and Computing, (89). https://doi.org/10.1145/1328567.1328572
Berke, L. (2017). Displaying confidence from imperfect automatic speech recognition for captioning. ACM SIGACCESS Accessibility and Computing, 118. https://doi.org/10.1145/3051519.3051522
Berke, L., Caulfield, C., & Huenerfauth, M. (2017). Deaf and hard-of-hearing perspectives on imperfect automatic speech recognition for captioning one-on-one meetings. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3132525.3132541
Berke, L., Kafle, S., & Huenerfauth, M. (2018). Methods for evaluation of imperfect captioning tools by deaf or hard-of-hearing users at different reading literacy levels. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3173665
Bhagwat, S. R., Deshmukh, M. S., Kachhioria, R., et al. (2025). SmartSignLearn. IGI Global. https://doi.org/10.4018/979-8-3693-7560-0.ch025
Butler, J., Trager, B., & Behm, B. (2019). Exploration of automatic speech recognition for deaf and hard of hearing students in higher education classes. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3308561.3353772
Cardinal, P., & Boulianne, G. (2009). Real-time correction of closed captions. In Proceedings of Interspeech 2009. https://doi.org/10.21437/interspeech.2009-443
Cardinal, P., Boulianne, G., Comeau, M., & Rouat, J. (2007). Real-time correction of closed captions. In Proceedings of the Annual Conference of the International Speech Communication Association. https://doi.org/10.3115/1557769.1557803
Fathallah, N., Bhole, M., & Staab, S. (2024). Empowering the deaf and hard of hearing community: Enhancing video captions using large language models. arXiv. https://doi.org/10.48550/arxiv.2412.00342
Gaur, Y., Metze, F., Miao, Y., & Reddy, C. (2015). Using keyword spotting to help humans correct captioning faster. In Proceedings of Interspeech 2015. https://doi.org/10.21437/interspeech.2015-595
Jain, D., Franz, R. L., Findlater, L., et al. (2018). Towards accessible conversations in a mobile context for people who are deaf and hard of hearing. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3234695.3236362
Kafle, S. (2019). Word importance modeling to enhance captions generated by automatic speech recognition for deaf and hard of hearing users.
Kafle, S., & Huenerfauth, M. (2016). Effect of speech recognition errors on text understandability for people who are deaf or hard of hearing. In Proceedings of the Workshop on Speech and Language Processing for Assistive Technologies (SLPAT). https://doi.org/10.21437/slpat.2016-4
Kang, J. J., Layton, E., Martin, D., et al. (2024). Towards improving real-time head-worn display caption mediated conversations with speaker feedback for hearing conversation partners. In Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613905.3650976
Lasecki, W. S., Miller, C. D., Kushalnagar, R. S., & Bigham, J. P. (2013). Legion scribe: Real-time captioning by non-experts. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/2461121.2461151
Lasecki, W. S., Kushalnagar, R., & Bigham, J. P. (2014). Legion scribe: Real-time captioning by non-experts. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility. https://doi.org/10.1145/2661334.2661352
Lasecki, W. S., Miller, C. D., Naim, I., et al. (2017). Scribe: Deep integration of human and machine intelligence to caption speech in real time. Communications of the ACM, 60(11), 93–100. https://doi.org/10.1145/3068663
Loizides, F., Basson, S. H., Kanevsky, D., et al. (2020). Breaking boundaries with live transcribe: Expanding use cases beyond standard captioning scenarios. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3373625.3417300
Paudyal, P., Banerjee, A., Hu, Y., et al. (2019). DAVEE: A deaf accessible virtual environment for education. In Proceedings of the 2019 ACM Conference. https://doi.org/10.1145/3325480.3326546
Romero-Fresco, P., & Fresno, N. (2023). The accuracy of automatic and human live captions in English. Linguistica Antverpiensia, New Series – Themes in Translation Studies.
Romriell, J. N., Brooksby, S. L., Roylance, S. A., et al. (2009). Methods and systems related to text caption error correction (Patent).
Ruiz-Arroyo, A., García-Crespo, Á., & Fuenmayor-Gonzalez, F. (2022). Comparative analysis between a respeaking captioning system and a captioning system without human intervention. Universal Access in the Information Society. https://doi.org/10.1007/s10209-022-00926-3
Seita, M., Lee, S., Andrew, S., et al. (2022). Remotely co-designing features for communication applications using automatic captioning with deaf and hearing pairs. In Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3441852.3476551
Simpson, M. N., Barrett, J., Bell, P., et al. (2016). Just-in-time prepared captioning for live transmissions. https://doi.org/10.1049/IBC.2016.0027
Takagi, H., Itoh, T., & Shinkawa, K. (2015). Evaluation of real-time captioning by machine recognition with human support. In Proceedings of the International Conference on Computers Helping People with Special Needs. https://doi.org/10.1145/2745555.2746648
Van Waes, L., Delbeke, T., Leijten, M., et al. (2009). Live-ondertiteling met spraakherkenning: een case-analyse van reductie en fouten.
Wald, M. (2006). Captioning for deaf and hard of hearing people by editing automatic speech recognition in real time. In Lecture Notes in Computer Science. https://doi.org/10.1007/11788713_100
Wald, M. (2006). Creating accessible educational multimedia through editing automatic speech recognition captioning in real time. Interactive Technology and Smart Education. https://doi.org/10.1108/17415650680000058
Wald, M., Bell, J.-M., & Boulain, P. (2007). Correcting automatic speech recognition captioning errors in real time. International Journal of Speech Technology. https://doi.org/10.1007/s10772-008-9014-4
Wald, M., & Bain, K. (2007). Enhancing the usability of real-time speech recognition captioning through personalised displays and real-time multiple speaker editing and annotation. In Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-540-73283-9_50
Widiartha, K. K., Agustini, K., & Tegeh, I. M. (2024). Real time automated speech recognition transcription and sign language character animation on learning media. Jurnal Nasional Pendidikan Teknik Informatika (JANAPATI). https://doi.org/10.23887/janapati.v13i3.85065
Yamamoto, K., Suzuki, I., Shitara, A., et al. (2021). See-through captions: Real-time captioning on transparent display for deaf and hard-of-hearing people. In Proceedings of the ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3441852.3476551
ZainEldin, H., Gamel, S. A., Talaat, F. M., et al. (2024). Silent no more: A comprehensive review of artificial intelligence, deep learning, and machine learning in facilitating deaf and mute communication. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10816-0
Zhang, J., Shiraishi, Y., Kumai, K., et al. (2016). Real-time captioning of sign language by groups of deaf and hard-of-hearing people. In Proceedings of the ACM International Conference on Interactive Systems. https://doi.org/10.1145/3011141.3011143
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Rahman Ari Putra, Nabila Putri Handayani, Dimas Arya Saputra, Intan Maharani Kusuma (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
