Filipino sign language recognition using deep learning

Myron Darrel Montefalcon, Jay Rhald Padilla, Ramon Llabanes Rodriguez

2021 5th International Conference on E-Society, E-Education and E-Technology, 219-

225, 2021

The Filipino deaf community continues to lag behind the fast-paced and technology-
driven society in the Philippines. The use of Filipino Sign Language (FSL) has
contributed to the improvement of communication of deaf people, however, the
majority of the population in the Philippines do not understand FSL. This project
utilized computer vision in obtaining the images and Convolutional Neural Network
(CNN) ResNet architecture in building the automated FSL recognition model, with the
goal of bridging the communication gap between the deaf community and the hearing
majorities. In the experimentation, the dataset used are static images generated from a
signer which gestured Filipino number signs which range from (0-9). Based on
experimentation, the best-achieved performance is on fine-tuned ResNet-50 model
which obtained a validation accuracy as high as 86.7% when the epoch value equals
15. For future work, real-time FSL recognition will be implemented and more data
will be collected to enable recognition of Filipino alphabets, basic phrases, and
common greetings.

Filipino Sign Language Recognition Using Long Short-Term

Memory and Residual Network Architecture
Myron Darrel Montefalcon, Jay Rhald Padilla, Ramon Rodriguez

Proceedings of Seventh International Congress on Information and Communication

Technology: ICICT 2022, London, Volume 4, 489-497, 2022

Filipino sign language (FSL) has improved communication among deaf people.
However, the majority of the population in the Philippines does not understand FSL.
This study explored computer vision in obtaining the data and deep learning in
training the continuous SLR model. The model has been trained using ResNet and
LSTM model using the features extracted using MediaPipe Holistic from video files
of Filipino phrases performed by three (3) FSL signers. The SLR system developed
can recognize (15) continuous Filipino phrases. Based on the comparison of both deep
learning models, the best performing model is from the LSTM setup, wherein the
accuracy achieved is 94%. In comparison, ResNet produced 87% accuracy on the test
set. The feature importance analysis has ranked the facial component in this order
(eyebrows, eyes, and mouth). The analysis shows that the exclusion of such features
has negatively affected the performance of the SLR model compared to the model
with full facial features. Based on the analysis of the experimentation results, the
developed SLR system is robust, time-efficient, signer-independent, and can detect
both manual and non-manual features of the gesture.

Sign Language Recognition of Selected Filipino Phrases Using

LSTM Neural Network
Myron Darrel Montefalcon, Jay Rhald Padilla, Ramon Rodriguez

Proceedings of Seventh International Congress on Information and Communication

Technology: ICICT 2022, London, Volume 4, 633-641, 2022

The use of Filipino Sign Language (FSL) has contributed to the improvement of
communication of deaf people; however, the majority of the population in the
Philippines does not understand FSL. The study explored computer vision in obtaining
the images and deep learning techniques in building the automated FSL recognition
model to bridge the communication gap between the deaf community and the hearing
majority. The model has been trained using LSTM neural network using the features
extracted using MediaPipe Holistic from video files of Filipino phrases performed by
three (3) FSL signers. The SLR system developed could recognize (15) continuous
Filipino words. The model evaluation has shown an impressive result wherein the
average accuracy achieved on the test set is 94%. In the experimentation conducted on
10 participants using the SLR system, the overall accuracy obtained on two trials is
72.38%, with an average prediction time of 0.3 s. Based on the analysis of the
experiment results, the developed SLR system is robust, is time efficient, is signer
independent, and can detect both manual and non-manual features of the gesture.
More data will be collected for future directions to enable a conversational SLR
system with more FSL vocabularies and sentences.

Sign Language Recognition of Selected Filipino Phrases Using

LSTM Neural Network
Myron Darrel Montefalcon, Jay Rhald Padilla, Ramon Rodriguez

Proceedings of Seventh International Congress on Information and Communication

Technology: ICICT 2022, London, Volume 4, 633-641, 2022

The use of Filipino Sign Language (FSL) has contributed to the improvement of
communication of deaf people; however, the majority of the population in the
Philippines does not understand FSL. The study explored computer vision in obtaining
the images and deep learning techniques in building the automated FSL recognition
model to bridge the communication gap between the deaf community and the hearing
majority. The model has been trained using LSTM neural network using the features
extracted using MediaPipe Holistic from video files of Filipino phrases performed by
three (3) FSL signers. The SLR system developed could recognize (15) continuous
Filipino words. The model evaluation has shown an impressive result wherein the
average accuracy achieved on the test set is 94%. In the experimentation conducted on
10 participants using the SLR system, the overall accuracy obtained on two trials is
72.38%, with an average prediction time of 0.3 s. Based on the analysis of the
experiment results, the developed SLR system is robust, is time efficient, is signer
independent, and can detect both manual and non-manual features of the gesture.
More data will be collected for future directions to enable a conversational SLR
system with more FSL vocabularies and sentences.

Understanding Facial Expression Expressing Hate from Online

Short-form Videos
Myron Darrel Montefalcon, Jay Rhald Padilla, Joshua Paulino, Jeline Go, Ramon
Llabanes Rodriguez, Joseph Marvin Imperial

2021 5th International Conference on E-Society, E-Education and E-Technology, 201-

207, 2021

The impact of hate speech is not only detrimental to an individual's human rights; but
also, a grave threat to social stability and democracy. Through social media, the
spread of hate speech has alarmingly increased across the globe. Various social media
platform's goal is to eliminate hateful content and this challenge poses the need for
automatic and accurate hate speech detection. Presently, known techniques in this
research primarily made use of either text or audio features. However, the use of the
facial expression in hate speech detection is not that explored. Thus, for this study, the
use of facial expressions to understand hate speech has been thoroughly investigated.
The dataset used is image data generated from Filipino Tiktok videos with a frame
size of 1080 x 1920 pixels and divided into 5 frames per second. Two approaches
namely conventional and deep learning-based frameworks have been implemented in
building the Facial Expression Recognition (FER) model to understand hate speech.
Based on the experimentation, the conventional approach using the Random Forest
approach has achieved the best performance with 86.9% training accuracy and 84.8%
validation accuracy, outperforming the other conventional classifiers and the DL-
based approach significantly. For future direction, facial expression features combined
with text or audio input type will be implemented to examine whether the use of facial
expression can complement or improve hate speech detection models.

iRespondPH: A mobile and web-based application for post

disaster needs assessment and response in the Philippines
Myron Darrel Montefalcon, Jay Rhald Padilla, Michael Ibanez, Ranz Sapinit, Lloyd
Lois Reyes, Ramon Llabanes Rodriguez, Elcid Serrano

2021 5th International Conference on E-Society, E-Education and E-Technology, 54-

61, 2021

The Philippines is one of the most hazard-prone countries in the world, from natural
disasters such as tropical cyclones, floods, earthquakes, drought, and volcanic
eruptions. Being stricken by the worst of the disasters it has frozen many
developments in the Philippines and it has negatively affected the people's health,
lives, properties, livelihood, and their living environment. As natural disasters are
unpredictable and inevitable, appropriate disaster risk reduction measures are
important, but the post-disaster response is as important and as essential, an efficient
post-disaster response could help save more lives and minimize the adverse impact of
the calamity. In this paper, IRespondPH, a mobile and web-based application for Post
Disaster Need Assessment (PDNA) and disaster response in the Philippines was
developed. It aims to provide citizens with real-time disaster information and help
disaster managers to make data-driven decisions in the aftermath of a disaster based
on the data reported by the citizens. The development of IRespondPH by the
utilization of the Agile Software Development cycle was discussed. The testing of the
mobile application has been carried out using Agile testing, a continuous process with
4 levels, from component testing to integration testing, to system testing and
acceptance testing. Based on the results from simulated data, the tests showed a
favorable result with mean scores ranging from 3.8 to 4.6, for all areas of a usability
test. It further shows that the developed system is ready for implementation.