Development of an automatic LSM recognition system for mobile devices
Keywords:
Mexican sign language, Automatic recognition system, Dynamic time warping, Mobile appAbstract
This paper explores the effectiveness of using mobile devices for the automatic recognition of both static and dynamic signs in Mexican Sign Language (LSM). This is due to the fact that most current works on LSM recognition typically focus on one type of sign (static or dynamic) and often involve specialized devices beyond the reach of the general population. To conduct this study, a solution was implemented, consisting of an Android application for capturing videos and sending them to a cloud-based web service hosting an automatic recognition system. The system employs the Dynamic Time Warping (DTW) algorithm to compare the features of the received video with those of a set of videos categorized by sign. It is noteworthy that this work is the first to use DTW for static signs in LSM. Finally, an experimental study of the system's performance was conducted using three mobile devices with different capabilities for both video capture and processing. Results indicate that, while camera quality affects recognition efficiency, it is not a decisive factor. Additionally, due to the video characterization method and LSM-specific attributes, the system tends to recognize static signs better than dynamic ones, even when using DTW.
References
[1] INEGI (2020). Población con discapacidad o limitación en la actividad cotidiana por entidad federativa y tipo de actividad realiza según sexo, 2020. https://www.inegi.org.mx/app/tabulados/interactivos/?pxq=Discapacidad_Discapacidad_02_2c111b6a-6152-40ce-bd39-6fab2c4908e3&idrt=151&opc=t
[2] Diario Oficial de la Federación (2005). Ley general de las personas con discapacidad, publicada el 10 de Junio de 2005, modificada el 1 de Agosto de 2008 y abrogada el 30 de Mayo de 2011. Diario Oficial de la Federación, Órgano del Gobierno Constitucional de los Estados Unidos Mexicanos. https://www.dof.gob.mx/nota_detalle.php?codigo=2044351&fecha=10/06/2005#gsc.tab=0
[3] Cruz-Aldrete, M. (2014). Hacia la construcción de un diccionario de lengua de señas mexicana. Revista de Investigación, 38(83), 57–80.
[4] Cámara de Diputados (2021). Aprueban reformas para que personas con discapacidad auditiva reciban educación bilingüe en lengua de señas. Boletín No. 5854 de la Cámara de Diputados del Congreso de la Unión de México.
[5] Pérez-Castro, J., & Cruz-Cruz, J.C. (2021). Inclusion-exclusion experiences of a group of deaf people users of the Mexican sign language. Revista Latinoamericana de Educación Inclusiva, 15(1), 39–54. https://dx.doi.org/10.4067/S0718-73782021000100039
[6] Flores-Saldaña, M.C., Cruz-Aldrete, M., Guajardo-Ramos, E., & Moreno-Aguirre, A.J. (2022). Percepción de inclusión educativa y atención a la salud de personas sordas en México. Revista ConCiencia EPG, 7(2), 16–29. https://doi.org/10.32654/CONCIENCIAEPG.7-2.2
[7] Ríos-Figueroa, H.V., Sánchez-García, A.J., Sosa-Jiménez, C.O., & Solís-González Cosío, A.L. (2022). Use of spherical and cartesian features for learning and recognition of the static Mexican sign language alphabet. Mathematics, 10(16), 2904. https://doi.org/10.3390/math10162904
[8] Rodríguez, R.F., Rosas, F.J.P., Zuñiga-Madrid, L.Á., & Arguijo, P. (2021). Reconocimiento de las señas estáticas del LSM con características basadas en aprendizaje profundo. Research in Computing Science, 150(6), 303–311.
[9] Fregoso, J., Gonzalez, C.I., & Martinez, G.E. (2021). Optimization of convolutional neural networks architectures using PSO for sign language recognition. Axioms, 10(3), 139. https://doi.org/10.3390/axioms10030139
[10] García-Gil, G., López-Armas, G.D.C., Sánchez-Escobar, J.J., Salazar-Torres, B.A., & Rodríguez-Vázquez, A.N. (2024). Real-time machine learning for accurate Mexican sign language identification: A distal phalanges approach. Technologies, 12(9), 152. https://doi.org/10.3390/technologies12090152
[11] Morfín-Chávez, R.F., Gortarez-Pelayo, J.J., & Lopez-Nava, I.H. (2024). Fingerspelling recognition in Mexican sign language (LSM) using machine learning. En: Calvo, H., Martínez-Villaseñor, L., Ponce, H. (eds) Advances in Computational Intelligence. MICAI 2023. Lecture Notes in Computer Science, vol. 14391, 110–120. Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-47765-2_9
[12] Espejel-Cabrera, J., Cervantes, J., García-Lamont, F., Ruiz Castilla, J.S., & Jalili, L.D. (2021). Mexican sign language segmentation using color based neuronal networks to detect the individual skin color. Expert Systems with Applications, 183, 115295. https://doi.org/10.1016/j.eswa.2021.115295
[13] Mejía-Peréz, K., Córdova-Esparza, D.M., Terven, J., Herrera-Navarro, A.M., García-Ramírez, T., & Ramírez-Pedraza, A. (2022). Automatic recognition of Mexican sign language using a depth camera and recurrent neural networks. Applied Sciences, 12(11), 5523. https://doi.org/10.3390/app12115523
[14] Trujillo-Romero, F., & Caballero-Morales, S.-O. (2013). 3D Data sensing for hand pose recognition. En: CONIELECOMP 2013, 23rd International Conference on Electronics, Communications and Computing, 109–113. https://doi.org/10.1109/CONIELECOMP.2013.6525769
[15] Solis-V, J.F., Toxqui-Quitl, C., Martínez-Martínez, D., & Margarita, H.G. (2014). Mexican sign language recognition using normalized moments and artificial neural networks. En: Awwal A.A.S, Iftekharuddin, K.M., Matin, M.A., Márquez, A. (eds), Optics and Photonics for Information Processing VIII, SPIE Proceedings, Vol. 9216, 92161A. https://doi.org/10.1117/12.2061077
[16] García-Bautista, G., Trujillo-Romero, F., & Caballero-Morales, S.O. (2017). Mexican sign language recognition using kinect and data time warping algorithm. En: CONIELECOMP 2017, 1–5. https://doi.org/10.1109/CONIELECOMP.2017.7891832
[17] Sosa-Jiménez, C.O., Ríos-Figueroa, H.V., & Solís-González-Cosío, A.L. (2022). A prototype for Mexican sign language recognition and synthesis in support of a primary care physician. IEEE Access, 10, 127620–127635. https://doi.org/10.1109/ACCESS.2022.3226696
[18] González-Rodríguez, J.R., Córdova-Esparza, D.M., Terven, J., & Romero-González, J.A. (2024). Towards a bidirectional Mexican sign language–spanish translation system: A deep learning approach. Technologies, 12(1), 7. https://doi.org/10.3390/technologies12010007
[19] Instituto Federal de Telecomunicaciones (2021). Encuesta nacional sobre disponibilidad y uso de tecnologías de la información en los hogares. https://www.inegi.org.mx/contenidos/saladeprensa/boletines/2021/OtrTemEcon/ENDUTIH_2020.pdf
[20] Martinez-Seis, B., Pichardo-Lagunas, O., Rodriguez-Aguilar, E., & Saucedo-Diaz, E.-R. (2019). Identification of static and dynamic signs of the Mexican sign language alphabet for smartphones using deep learning and image processing. Research in Computing Science, 148(11), 199–211.
[21] Pham, C.H., Le, Q.K., & Le, T.H. (2014). Human action recognition using dynamic time warping and voting algorithm. VNU Journal of Science: Computer Science and Communication Engineering, 30(3).
[22] Guerin, G. (2021). Sign language recognition - using mediapipe dtw. https://www.sicara.fr/blog-technique/sign-language-recognition-using-mediapipe
[23] Howse, J., & Minichino, J. (2020). Learning OpenCV 4 computer vision with Python 3. Packt Publishing Ltd.
[24] CloudRun (2023). ¿Qué es cloud run? https://cloud.google.com/run/docs/overview/what-is-cloud-run?hl=es-419
[25] MediaPipe (2023). Mediapipe solutions guide. https://developers.google.com/mediapipe/
[26] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019). Deep learning for time series classification: A review. Data Mining and Knowledge Discovery, 33(4), 917–963. https://doi.org/10.1007/s10618-019-00619-1
[27] (2007). Dynamic time warping. En: Information Retrieval for Music and Motion. Springer, Berlin, Heidelberg, 69–84. https://doi.org/10.1007/978-3-540-74048-3_4
[28] Salvador, S. & Chan, P. (2004). FastDTW: Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis, 11(5), 561–580. https://doi.org/10.3233/IDA-2007-11508
[29] Kruskal, W.H., & Wallis, W.A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. https://doi.org/10.2307/2280779
[30] Dunn, O.J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3), 241–252. https://doi.org/10.2307/1266041
[31] Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.2307/2331554
